SQL is a powerful language. It is a declarative language where we define what we want and the engine finds a way to achieve it. The language is evolving to adapt to the increasing variety of analysis needs. I wrote an article about an advanced SQL feature to deal with hierarchical data. And today, let’s explore another beyond-the-basic feature: PIVOT.

Problem Statement

Imagine you are working as a data engineer for a retail company. The company wants to analyze product sales data to identify trends and opportunities for growth. The data is stored in a table called Sales with the following structure:

This structure is not good for reports. The company wants this data served in a format where years are represented as columns for easier comparison across products.

GROUP BY - The Amateur Way

A very straightforward approach to this problem is to use the GROUP BY statement. We will group by ProductID, and we will get the sum column for each month. Below is a SQL Server example. Other SQL engines should have similar syntax.

select
    ProductID
    ,sum(iif(year(Date)=2023, Amount, null)) as [2023]
    ,sum(iif(year(Date)=2024, Amount, null)) as [2024]
    ,sum(iif(year(Date)=2025, Amount, null)) as [2025]
from Sales
group by ProductID;

At first glance, it’s simple, and it works. Sometimes just working is enough.

SQL PIVOT - The Complex Way

PIVOT is a operator in SQL that allows you to transform rows into columns. This transformation is particularly useful when summarizing data and creating a more interpretable format for analysis. The below SQL Server query achieves similar results:

select 
    *
from (
    select 
        ProductID, 
        year(Date) as Year, 
        Amount 
    from Sales
) as ToPivotSales pivot (
    sum(Amount) for Year in ([2023], [2024], [2025])
) as PivotedSales

To do PIVOT, we first need a subquery to specify the columns we need. In this case they are ProductID, Year and Amount. If you don’t like subqueries, CTE (Common Table Expression) works as well.

In the PIVOT decalaration:

• sum(Amount): specifies that the aggregation function to be applied is sum, which will sum the Amount values.
• for Year in ([2023], [2024], [2025]): defines how the pivoting will occur:
  • for Year: indicates that the values in the Year column will be used to create new columns in the result set.
  • in ([2023], [2024], [2025]): specifies the specific years that will become the new columns in the result. Each of these years will have a corresponding column that contains the summed Amount for that year.

To be honest, I don’t like the syntax of PIVOT. It is terrible to me. It involves subqueries and CTEs, it uses more levels of indentation, and it’s harder to scan through. And if we look at the execution plan, it is not really faster than GROUP BY.

With that being said, PIVOT still has advantages over GROUP BY that it requires less boilerplate code. In the examples above, to add a new year to the query with GROUP BY, you have to copy, paste, and edit in 2 places. With PIVOT, all you have to do is add a new value to the list. This makes PIVOT shine in situations where we have to deal with a long list of values.

Dynamic PIVOT - The Hacker Way

While both GROUP BY and PIVOT are useful, they have the limitation that you must explicitly specify the list of values. When the data is small, this is fine. But it becomes a problem when dealing with evolving, large data where you do not know or do not want to manually list all possible values. Imagine you build a report of annual sales, you clearly don’t want to update the query every year.

Dynamic PIVOT is a technique that allows us to pivot data without hard-coding the pivot column values. It is not a standard SQL operation. Therefore, the syntax may vary between different SQL engines. In Snowflake SQL, you can achieve dynamic pivoting by something as simple as this.

...
pivot (
    sum(Amount) for Year in (any order by Year)
)

Most other SQL engines don’t offer this level of simplicity. They require a bit of complexity. Below is a SQL Server query example.

declare
    @cols as nvarchar(max),
    @query as nvarchar(max);


select @cols = string_agg(quotename(Year), ',') within group (order by Year)
from (select distinct year(Date) as Year from Sales) as YearList;


set @query = (
    'select *
    from (
        select ProductID, year(Date) as Year, Amount
        from Sales
    ) as ToPivotSales pivot (
        sum(Amount) for Year in (' + @cols + ')
    ) as PivotedSales;'
);


exec sp_executesql @query;

Although the code is not pleasing to eyes, the concept is simple. We need an additional step to find all the unique values and store them in a variable. Then we construct the query with the above list of unique values. Execute it and we get the expected result.

Final Thought

Mastering SQL is not easy. SQL is more than just a query language; it’s a powerful tool that, when used effectively, can turn raw data into actionable intelligence. Advanced SQL techniques such as PIVOT are not very common in our day-to-day work. But we should know what we have in our toolbox. So that we can quickly select the right tool from the toolbox where the right job applies.

* You can find the execution of SQL query examples in this post at https://dbfiddle.uk/2jD1lHkL

Hierarchical Data

Hierarchical data represents a natural parent-child relationship that is often visualized in the form of a tree structure. Imagine a family tree: grandparents on top, parents in the middle, and you and siblings at the bottom, all connected. That’s hierarchical data! It organizes information in levels, making it easy to understand how things are related. The most popular real-life example of hierarchical data is employee-manager relationships. Every employee is managed by a manager. And that manager is also an employee and (again) is managed by another manager who is also an employee.

Hierarchical Data representation in SQL

Relational models perform best with flat tables with rows and columns, not tree-like structures. However, people developed techniques to represent hierarchical data in SQL. The most common approach is referencing using foreign key. In the above example, we will add a column manager_id referring to the person who managed this employee to the employee table.

*Example SQL table representing hierarchical data structure

Querying Hierarchical Data

When querying hierarchical data, we often want to understand the relationship in both directions: who manages whom and who is managed by whom. However, querying hierarchical data is tricky because we don’t know the depth of the tree, i.e. how many levels of hierarchy there are. Before we look at how to do this in SQL, let’s prepare some data to work with. Note that all SQL code in this post is Oracle, as it natively supports CONNECT BY. Other RDBMS SQL should be similar.

CREATE TABLE EMPLOYEES (
  EMPLOYEE_ID NUMBER PRIMARY KEY,
  NAME VARCHAR2(50) NOT NULL,
  SALARY NUMBER,
  MANAGER_ID NUMBER REFERENCES EMPLOYEES(EMPLOYEE_ID)
);

INSERT INTO EMPLOYEES VALUES
(1, 'Adam', 60000, NULL),
(2, 'Sarah', 70000, NULL),
(3, 'David', 50000, 1),
(4, 'Emily', 55000, 2),
(5, 'Michael', 45000, 1),
(6, 'Jessica', 50000, 2),
(7, 'Ben', 35000, 3),
(8, 'Olivia', 37000, 4),
(9, 'Charles', 32000, 5),
(10, 'Sophia', 33000, 6),
(11, 'Alex', 37000, 3),
(12, 'Maya', 38000, 4),
(13, 'Daniel', 35000, 5),
(14, 'Isabella', 36000, 6),
(15, 'Ryan', 25000, 7),
(16, 'Chloe', 26000, 8),
(17, 'Noah', 24000, 9),
(18, 'Mia', 25000, 10),
(19, 'Liam', 26000, 11),
(20, 'Evelyn', 27000, 12),
(21, 'William', 25000, 13),
(22, 'Charlotte', 26000, 14),
(23, 'Ethan', 27000, 7),
(24, 'Ava', 28000, 8),
(25, 'Lucas', 26000, 9),
(26, 'Amelia', 27000, 10),
(27, 'Mason', 28000, 11),
(28, 'Harper', 29000, 12),
(29, 'Logan', 27000, 13),
(30, 'Sofia', 28000, 14);

Problem statement

We want to look at it from two directions:

• Problem 1: Start with individual employees and follow the ladder, revealing who manages them, their manager’s manager, and so on, all the way to the top.
• Problem 2: Stand at the highest level and look down the hierarchy. For each employee, calculate the total salary of everyone under their direct or indirect management, like a salary pyramid.

When looking at direct relationships (who manages who), a simple join clause will work. But things get tricky when we include multiple levels in the hierarchy.

Recursive CTE

A recursive CTE (Common Table Expression) is a valuable feature in SQL. By referencing itself within the CTE, it allows you to query hierarchical data by repeating a process until it reaches every corner, making it an effective tool for querying and analyzing hierarchical data.

The recursive CTE syntax is not too different from non-recursive one.

Non-recursive CTE:

WITH CTE_NAME (column_1, column2, ...) AS 
(
    -- CTE_QUERY_DEFINITION (SELECT ... FROM ... WHERE)
)

Recursive CTE:

WITH CTE_NAME (column_1, column2, ...) AS
(
    -- ANCHOR_MEMBER (SELECT ... FROM ... WHERE BASE_LEVEL)
    
    UNION (ALL)
    
    -- RECURSIVE_MEMBER (SELECT ... FROM (referrence to CTE_NAME) WHERE ...)
)

The definition of a recursive CTE consists of two parts. The anchor member or the initial query, which is executed once. This defines the starting point of the execution. The recursive query has the reference to the CTE itself and is executed recursively until it returns no result. And we need UNION or UNION ALL to combine the results. You will see how it work when use it to solve real problems.

Problem 1

For each employee, we get the direct manager, the path from the highest-level manager, and the employee’s current level.

WITH company_hierarchy (EMPLOYEE_ID, NAME, MANAGER, MANAGER_PATH, EMPLOYEE_LEVEL) AS
(
    -- Base query. Select ALL employees with no manager
    SELECT
        emp.EMPLOYEE_ID
        ,emp.NAME
        ,NULL AS MANAGER
        ,'' AS MANAGER_PATH
        ,1 AS EMPLOYEE_LEVEL
    FROM EMPLOYEES emp
    WHERE emp.MANAGER_ID IS NULL
    
    UNION ALL
    
    -- Recursive query which refer to the CTE itself.
    -- Query all employee who is managed directly by previous level in company_hierarchy
    SELECT
        emp.EMPLOYEE_ID
        ,emp.NAME
        ,com.NAME AS MANAGER
        ,com.MANAGER_PATH || com.NAME || '/' AS MANAGER_PATH
        ,com.EMPLOYEE_LEVEL + 1 AS EMPLOYEE_LEVEL 
    FROM EMPLOYEES emp
    INNER JOIN company_hierarchy com
        ON emp.MANAGER_ID = com.EMPLOYEE_ID    
)

SELECT *
FROM company_hierarchy

The below picture illustrates how the recursive CTE works:

First, the anchor member or initial query was executed. company_hierarchy CTE contained 2 members with MANAGER_ID IS NULL and EMPLOYEE_LEVEL=1.

Then the recursive query was executed, joining the EMPLOYEES table with company_hierarchy to get all members managed by 2 employees currently in company_hierarchy. company_hierarchy was then set to all returned employees (EMPLOYEE_LEVEL=2).

We kept doing so until the recursive query returned nothing.

At the end, we UNION ALL all the results.

Problem 2

This is a bit more difficult. Let’s reread the problem statement and try yourself before reading my answer.

WITH company_explode (MANAGER_ID, EMPLOYEE_ID, SALARY, MANAGEMENT_DISTANCE) AS
(
    -- Base query. Select ALL employees
    SELECT 
        emp.MANAGER_ID
        ,emp.EMPLOYEE_ID
        ,emp.SALARY
        ,1 AS MANAGEMENT_DISTANCE
    FROM EMPLOYEES emp
    
    UNION ALL
    
    -- Recursive query. Select ALL managers manages the employees
    SELECT
        mgr.MANAGER_ID
        ,com.EMPLOYEE_ID
        ,com.SALARY
        ,com.MANAGEMENT_DISTANCE+1 AS MANAGEMENT_DISTANCE
    FROM EMPLOYEES mgr
    INNER JOIN company_explode com
        ON mgr.EMPLOYEE_ID = com.MANAGER_ID
)

SELECT MANAGER_ID, SUM(SALARY) AS TOTAL_SALARY_MANAGED
FROM company_explode
GROUP BY MANAGER_ID

The idea behind this solution is simple. First, explode the hierarchical structure of the data into a flattened table of all manager-employee pairs. Then calculate the sum with a simple GROUP BY clause.

START WITH … CONNECT BY …

CONNECT BY clause achieves similar functionality as a recursive Common Table Expression (CTE), but with a much shorter syntax. Let’s take a look at it in action.

Problem 1

SELECT
    emp.EMPLOYEE_ID
    ,emp.NAME
    ,mgr.NAME AS MANAGER
    ,TRIM(LEADING '/' FROM SYS_CONNECT_BY_PATH(mgr.NAME, '/')) AS MANAGER_PATH
    ,LEVEL AS EMPLOYEE_LEVEL
FROM EMPLOYEES emp
LEFT JOIN EMPLOYEES mgr
    ON emp.MANAGER_ID = mgr.EMPLOYEE_ID
START WITH emp.MANAGER_ID IS NULL
CONNECT BY PRIOR emp.EMPLOYEE_ID = emp.MANAGER_ID;

Problem 2

SELECT
    emp.MANAGER_ID
    ,SUM(CONNECT_BY_ROOT emp.SALARY) AS TOTAL_SALARY_MANAGED
FROM EMPLOYEES emp
CONNECT BY PRIOR emp.MANAGER_ID = emp.EMPLOYEE_ID
GROUP BY emp.MANAGER_ID

Like magic, isn’t it! The implementation details (what goes on behind the scenes) may be a bit different. But the way we reason about the query is the same as it is with Recursive CTEs. The START WITH clause provides the initial filter. The query is first executed with this filter, just like the anchor member in recursive CTEs. No START WITH clause means no filter is needed and the entire query is included in the first step. Then the CONNECT BY clauses specify how to connect between steps/levels in the hierarchical structure, just like recursive CTEs refer to themselves. Note that the PRIOR keyword implies that the following value is from the previous recursive step.

One of the main differences between the two approaches is how we select data. With CONNECT BY, we have to rely on built-in functions and operators to select the data we want. In the examples, we use the SYS_CONNECT_BY_PATH function to construct the path and the CONNECT_BY_ROOT operator to access the data in the first step.

My final thoughts

Querying hierarchical data presents unique challenges that require specialized techniques to extract meaningful information. Recursive CTEs and the CONNECT BY clause offer powerful solutions for navigating and analyzing hierarchical data in SQL. One interesting fact is that CONNECT BY was actually around before Recursive CTEs.

While both techniques solve the same problem, we can only use one. Which one should you use? Well, it depends on you. If you hate subqueries and CTEs, and you like cool short magic queries, use CONNECT BY. However, being less verbose makes CONNECT BY harder to reason about. It’s harder to write out that magic stuff, and harder to know why it doesn’t work. Also, by writing the recursion yourself, recursive CTEs give you more control and flexibility. And note that not all RDBMS (SQL Server for example) support the CONNECT BY clause, even though it has been around for a long time.

* You can find the execution of SQL in this post at https://dbfiddle.uk/hXEzBJkX