The dataset

I’ll be using sales funnel data. In business, some sales cycles are very long; for instance, those involving enterprise software or capital equipment, and management often wants to understand them in more detail throughout the year. The following are some of the questions they are likely to ask:

1. How much revenue is in the pipeline?
2. What products are in the pipeline?
3. Who has which products at what stage?
4. How likely are we to close deals by year-end?

Larger companies typically have CRM tools or other software that the sales team uses to track the sales process. Still, many people simply export the data to Excel, where they summarize it by creating a pivot table.

The benefits of creating a pivot table with Polars are:

• It’s quicker once it’s set up.
• It’s self-documenting. You can look at the code and understand what it does.
• It’s easier to generate multiple reports.
• It offers the flexibility to define custom aggregation functions.

Let’s read in the data.

import polars as pl
pl.Config(set_tbl_rows=20)

df = pl.read_excel('https://pbpython.com/extras/sales-funnel.xlsx')
df

Creating pivot tables

The pivot method is used to create a pivot table in Polars. However, not all data outputs that look like pivot tables are created with pivot. Sometimes, group_by does the trick.

Suppose we wanted to find the average (mean) Account, Price, and Quantity for each Name. We can do so using group_by.

(df
 .group_by('Name')
 .agg(pl.mean('Account','Price','Quantity'))
 )

In this case, the values in Name are unique keys that appear only once. If the original dataframe contained one name appearing two or more times, Polars would calculate the average of its values and display the company name only once in the final output. This is exactly what group_by does.

We can also group_by multiple columns.

(df
 .group_by('Name','Rep','Manager')
 .agg(pl.mean('Account','Price','Quantity'))
 )

Of course, this is not a particularly useful output. A better result is one that shows a combination of manager and representative as the key values.

(df
 .group_by('Manager','Rep')
 .agg(pl.mean('Account','Price','Quantity'))
 )

The output above shows that Fred Anderson has two representatives, while Debra Henley has three.

We can also perform an aggregation on a single column. The Account and Quantity columns aren’t providing useful values here, so I’ll exclude them.

(df
 .group_by('Manager','Rep')
 .agg(pl.mean('Price'))
 )

We can also use an aggregation other than mean.

(df
 .group_by('Manager','Rep')
 .agg(pl.sum('Price'))
 )

Or even use multiple aggregation functions.

(df
 .group_by('Manager','Rep')
1 .agg(pl.mean('Price').name.prefix('Avg_'),
      pl.count('Price').name.prefix('Count_'))
 )

Sometimes, we may want to see sales broken down by product. This is where pivot comes in. The values contained in the column specified by the on parameter become new columns in the final output.

(df
 .pivot(index=['Manager','Rep'],
        on='Product', values='Price',
        aggregate_function='sum')
 )

We can also use multiple columns in the values parameter.

(df
 .pivot(index=['Manager','Rep'],
        on='Product', values=['Price','Quantity'],
        aggregate_function='sum')
 )

A rule of thumb I follow is that whenever you want the values in a column to become new columns in the final output, use pivot; otherwise, use group_by. So, if we want Product to be part of the index and don’t need a column for the on parameter, then we’ll have to use group_by.

(df
 .group_by('Manager','Rep','Product')
 .agg(pl.sum('Price','Quantity'))
 .sort('Manager','Rep','Product')
 )

This output makes much more sense, at least for this dataset. Now, what if we want to include a grand total row at the bottom of the dataframe, much like you would see in an Excel pivot table? In pandas, this is easy to achieve. The pivot_table function has a margins=True parameter that adds a grand total row.

By contrast, Polars does not have a margins=True parameter in its pivot method. However, the versatility of Polars makes it possible to create a grand total row yourself.

(df
 .group_by('Manager','Rep','Product')
 .agg(pl.sum('Price','Quantity').name.prefix('Sum_'),
      pl.mean('Price','Quantity').name.prefix('Avg_'))
 .sort('Manager','Rep','Product')
 .pipe(lambda _df: _df.vstack(
       _df.select(Manager=pl.lit('All'),
                  Rep=pl.lit(None, dtype=pl.String),
                  Product=pl.lit(None, dtype=pl.String),
                  Sum_Price=pl.sum('Sum_Price'),
                  Sum_Quantity=pl.sum('Sum_Quantity'),
                  Avg_Price=pl.mean('Avg_Price'),
                  Avg_Quantity=pl.mean('Avg_Quantity'),
                  )))
)

Let’s look at the sales funnel at the manager level. At the moment, the Status column has the string data type. We’ll convert it to the enum data type and define the hierarchy in the order we want.

(df
 .with_columns(pl.col('Status').cast(pl.Enum(['won','pending','presented','declined'])))
 .group_by('Manager','Status')
 .agg(pl.sum('Price'))
1 .sort('Manager','Status', descending=[False,True])
 .pipe(lambda _df: _df.vstack(
     _df.select(Manager=pl.lit('All'),
                Status=pl.lit(None),
                Price=pl.sum('Price'),
                )))
 )

So far, we’ve used group_by and pivot separately. Once again, unlike pandas, where you can pass different aggregation functions for a single column or multiple columns in one pivot table call, Polars allows only one aggregation function at a time. However, the same result can be achieved by combining the group_by and pivot methods.

(df
 .group_by('Manager', 'Status', 'Product')
 .agg(Quantity=pl.len(),
      Price=pl.sum('Price'))
 .pivot(index=['Manager','Status'], on='Product',
        values=['Quantity','Price'], sort_columns=True)
 .sort('Manager','Status')
 .fill_null(0)
)

We can also perform more than two aggregations.

df_pivot_table = (df
 .group_by('Manager', 'Status', 'Product')
 .agg(Quantity=pl.len(),
1      Sum_Price=pl.sum('Price'),
      Avg_Price=pl.mean('Price'))
 .pivot(index=['Manager','Status'], on='Product',
        values=['Quantity','Sum_Price','Avg_Price'], sort_columns=True)
 .sort('Manager','Status')
 .fill_null(0)
)
df_pivot_table

Advanced pivot table filtering

Interestingly, we can perform filter operations on df_pivot_table because, despite being a pivot table, it’s still a regular dataframe.

Let’s filter the rows for a single manager, Debra Henley.

(df_pivot_table
 .filter(Manager='Debra Henley')
 )

Or we can look at the pending and won deals.

(df_pivot_table
 .filter(pl.col('Status').is_in(['pending','won']))
 )

Pivot tables are too useful and too powerful to remain the preserve of Microsoft Excel alone. They can be created with many data analysis tools, including Polars, pandas, and DuckDB.

Check out my Polars book to learn more about this powerful data analysis tool.

• Deep Analysis with Polars, on Leanpub

• Deep Analysis with Polars, on Amazon

What is the 52-week high?

The 52-week high is the highest price a particular stock has reached during the most recent 52-week period. The 52-week low represents the lowest price for that stock during the same period. Once you know these prices, you can gauge where the stock currently stands relative to where it has been recently. These levels are widely watched because they provide quick insight into a stock’s price range, market sentiment, and potential opportunities or risks.

Why care about the 52-week high/low?

If watching these two numbers closely can provide insight into potential opportunities and risks, then it’s important for a new investor like myself to understand what to look for and how to interpret what I see. It turns out there are four things to pay attention to when analyzing a stock’s 52-week high and low.

1. They show the stock’s trading range – The range between the high and the low tells you how volatile the stock has been. A wide range indicates higher volatility and larger price swings, while a narrow range suggests greater stability. For instance, Tesla (TSLA) often has a wide range because of growth expectations and sensitivity to news. By contrast, Coca-Cola (KO) tends to have a narrower range because of its steady business performance.

import yfinance as yf
import polars as pl
import polars.selectors as cs
import hvplot.polars
import holoviews as hv
from bokeh.themes import Theme
from bokeh.models import CustomJSTickFormatter, Div
from bokeh.layouts import column
from bokeh.io import output_notebook, show
import base64
from pathlib import Path
import datetime


# Define stock tickers
tickers = [
    "NVDA",  # Nvidia
    "TSLA",  # Tesla
    "KO",    # Coca-Cola
    "META",  # Meta
    "GME",   # GameStop
]

# Define date range
start_date = "2021-01-01"
end_date = "2026-06-11"

# Download data
data = yf.download(tickers, start=start_date, end=end_date, progress=False )

df = (pl.from_pandas(data.reset_index())
      .filter(pl.col("('Date', '')").dt.year().ne(2021))
      )

# Function to select specific stock
def select_ticker(df, ticker: str):
    return (df
        .select(cs.contains(ticker), "('Date', '')")
        .rename(lambda c: c.replace(f"', '{ticker}')","").replace("('","").replace("', '')",''))
        .select('Date', pl.all().exclude('Date'))
        .with_columns(pl.col('Date').cast(pl.Date))
        .with_columns(pl.col('High').rolling_max_by(by='Date', window_size='52w').alias('52-week High'),
                      pl.col('Low').rolling_min_by(by='Date', window_size='52w').alias('52-week Low'))
    )


font = 'Harmonia Sans Pro Cyr'
color_honeydew = '#F0FFF0'

hv.renderer('bokeh').theme = Theme(json={
    'attrs': {
        'Plot': {
            'background_fill_color': color_honeydew,
            'border_fill_color': color_honeydew,
            'outline_line_color': color_honeydew,
        },
        'Legend': {
            'background_fill_color': color_honeydew,
            'border_line_color': color_honeydew,
            'label_text_font': font,
            'label_text_font_size': '13pt',
        },
        'Text': {'text_font': font},
        'Axis': {
            'major_label_text_font': font,
            'axis_label_text_font': font,
            'major_label_text_font_size': '15pt',
        },
        'Title': {'text_font': font, 'text_font_size': '18pt', 'align': 'center'},
    }
})

logo_path = Path('../../images/logo_financical.png')
logo_b64 = base64.b64encode(logo_path.read_bytes()).decode('utf-8')
logo_data_uri = f"data:image/png;base64,{logo_b64}"

LOGO_H = 30 #change image size

def format_y_axis(bokeh_plot, element):
    p = bokeh_plot.state
    p.min_border_top = 60
    p.min_border_bottom = LOGO_H + 50  # space below x-axis for the logo

    top_tick = p.y_range.end
    p.yaxis.formatter = CustomJSTickFormatter(
        args={'top_tick': top_tick},
        code='''
            if (Math.abs(tick - top_tick) < (top_tick * 0.05)) {
                return '$' + tick.toFixed(0);
            }
            return tick.toFixed(0);
        '''
    )

# Create the chart
plot_tesla = (df
    .pipe(select_ticker, ticker='TSLA')
    .hvplot.line(x='Date', y=['52-week High', '52-week Low'],
                 title='Tesla stock 52-week high/low',
                 xlabel=' ', ylabel=' ', height=300, width=500,
                 legend_cols=1, legend='top_left', legend_opts={'title':''})
).opts(hooks=[format_y_axis])

plot_coke = (df
    .pipe(select_ticker, ticker='KO')
    .hvplot.line(x='Date', y=['52-week High', '52-week Low'],
                 title='Coca Cola stock 52-week high/low',
                 xlabel=' ', ylabel=' ', height=300, width=500,
                 legend_cols=1, legend='top_left', legend_opts={'title':''})
).opts(hooks=[format_y_axis])

combined = plot_tesla + plot_coke

bokeh_plot = hv.render(combined, backend='bokeh')

for item in bokeh_plot.children:
    p = item[0] if isinstance(item, tuple) else item
    p.title.text_font = font
    p.title.text_font_size = '18pt'
    p.title.align = 'left'
    p.background_fill_color = color_honeydew
    p.border_fill_color = color_honeydew
    p.outline_line_color = color_honeydew
    p.legend.background_fill_color = color_honeydew
    p.legend.border_line_color = color_honeydew

total_width = sum((item[0] if isinstance(item, tuple) else item).width for item in bokeh_plot.children)

logo_div = Div(
    text=f'''
        <div style="
            width: {total_width}px;
            height: {LOGO_H + 40}px;
            position: relative;
            top: -{LOGO_H + 40}px;
            display: flex;
            align-items: center;
            justify-content: flex-end;
            padding-right: 40px;
            background-color: transparent;
            pointer-events: none;
        ">
            <img src="{logo_data_uri}" height="{LOGO_H}px"/>
        </div>
    '''
)

final = column(bokeh_plot, logo_div, spacing=0)
show(final)

2. They indicate investor sentiment – A stock price near its 52-week high usually indicates strong bullish sentiment. A stock trading near its 52-week low often signals bearish sentiment. Investors can use this information to understand what the market thinks about a company. To an investor, the message coming from the market is often more reliable than the message coming from some stock-picking guru on television, as investor and entrepreneur Naval Ravikant advises:

Take feedback from nature and markets, not from people.

3. They act as psychological support and resistance levels – Investors often use these two values as important thresholds. The 52-week high can act as a resistance level, where investors may be inclined to sell because they expect a pullback. The 52-week low can act as a support level, where investors may buy because they expect a rebound. For example, if Apple (AAPL) approaches its 52-week high, traders might expect short-term resistance. However, if it approaches its 52-week low, value buyers may step in and see it as an opportunity.

4. They help identify potential buying or selling opportunities – Value investors often look for stocks near their 52-week low and consider buying them. As someone new to investing, this initially did not make sense to me. Why would someone want to buy a stock whose price is falling? However, this strategy makes sense to value investors because they believe the stock may be undervalued. Rather than being alarmed by the falling price, they see an opportunity to buy the stock at a bargain. A good example is Meta (META) in 2022. As shown in the shaded gray area on the chart, the stock traded near its 52-week low because of concerns about the company’s future growth. However, long-term investors who bought the stock during that period were later rewarded with significant gains when the company recovered.

chart = (df
 .pipe(select_ticker, 'META')
 .hvplot.line(x='Date', y=['52-week High', 'Close', '52-week Low'],
                 title="Meta's stock struggled in 2022",
                 xlabel=' ', ylabel=' ', height=400,
                 color=['#ff7f0e', '#1f77b4', '#d62728'],
                 legend_cols=1, legend='top_left', legend_opts={'title':''})
).opts(hooks=[format_y_axis])

highlight = hv.Rectangles([(datetime.date(2022, 1, 20), 100, datetime.date(2022, 11, 5), 400)]).opts(
    color='#1a1a1a', alpha=0.5, line_width=0
)

annotation = hv.Text(datetime.date(2023, 1, 15), 350, '2023 Start').opts(
    text_align='left', text_color='gray', text_font_size='10pt'
)

combined = chart * highlight * annotation

bokeh_plot = hv.render(combined, backend='bokeh')
bokeh_plot.title.text_font = font
bokeh_plot.title.text_font_size = '18pt'
bokeh_plot.title.align = 'left'
bokeh_plot.background_fill_color = color_honeydew
bokeh_plot.border_fill_color = color_honeydew
bokeh_plot.outline_line_color = color_honeydew
bokeh_plot.legend.background_fill_color = color_honeydew
bokeh_plot.legend.border_line_color = color_honeydew

logo_div = Div(
    text=f'''
        <div style="
            width: {bokeh_plot.width or 700}px;
            height: {LOGO_H + 40}px;
            position: relative;
            top: -{LOGO_H + 40}px;
            display: flex;
            align-items: center;
            justify-content: flex-end;
            padding-right: 40px;
            background-color: transparent;
            pointer-events: none;
        ">
            <img src="{logo_data_uri}" height="{LOGO_H}px"/>
        </div>
    '''
)

final = column(bokeh_plot, logo_div, spacing=0)
show(final)

As for selling opportunities, stocks trading near their 52-week highs may be overvalued or due for a correction. A good example is GameStop (GME) in 2021. The stock rapidly rose and hit new highs, but that did not mean it was fundamentally strong. In fact, it was highly risky. Many people who bought the stock believing the price would continue to rise eventually lost money. Clearly, GameStop had become overvalued, and the market eventually corrected the price.

plot_gamestop = (pl.from_pandas(data.reset_index())
    .pipe(select_ticker, ticker='GME')
    .hvplot.line(x='Date', y=['Close'],
                 title='GameStop stock fluctuations',
                 xlabel=' ', ylabel=' ', height=400,
                 legend_cols=1, legend='top_left', legend_opts={'title':''})
).opts(hooks=[format_y_axis])

font = 'Harmonia Sans Pro Cyr'

bokeh_plot = hv.render(plot_gamestop, backend='bokeh')
bokeh_plot.title.text_font = font
bokeh_plot.title.text_font_size = '18pt'
bokeh_plot.title.align = 'left'
bokeh_plot.background_fill_color = color_honeydew
bokeh_plot.border_fill_color = color_honeydew
bokeh_plot.outline_line_color = color_honeydew

logo_div = Div(
    text=f'''
        <div style="
            width: {bokeh_plot.width or 700}px;
            height: {LOGO_H + 40}px;
            position: relative;
            top: -{LOGO_H + 40}px;
            display: flex;
            align-items: center;
            justify-content: flex-end;
            padding-right: 40px;
            background-color: transparent;
            pointer-events: none;
        ">
            <img src="{logo_data_uri}" height="{LOGO_H}px"/>
        </div>
    '''
)

final = column(bokeh_plot, logo_div, spacing=0)
show(final)

Practical analysis of 52-week high/low

I downloaded data for a stock that is currently popular: NVIDIA. Using data from Yahoo Finance covering January 1, 2022, to June 11, 2026, I plotted NVIDIA’s 52-week high and low against its closing price to visualize the trend. Below is the chart showing that trend.

plot = (df
    .pipe(select_ticker, 'NVDA')
    .hvplot.line(x='Date', y=['52-week High', 'Close', '52-week Low'],
                 title='NVIDIA stock 52-week high/low',
                 xlabel=' ', ylabel=' ', height=400,
                 color=['#ff7f0e', '#1f77b4', '#d62728'],
                 legend_cols=1, legend='top_left', legend_opts={'title':''})
).opts(hooks=[format_y_axis])

# Render to Bokeh and fix title font (hv.render can drop theme title settings)
# Render to Bokeh and fix title font (hv.render can drop theme title settings)
bokeh_plot = hv.render(plot, backend='bokeh')
bokeh_plot.title.text_font = font
bokeh_plot.title.text_font_size = '18pt'
bokeh_plot.title.align = 'left'

# Explicitly re-apply background colors after hv.render()
bokeh_plot.background_fill_color = color_honeydew
bokeh_plot.border_fill_color = color_honeydew
bokeh_plot.outline_line_color = color_honeydew
bokeh_plot.legend.background_fill_color = color_honeydew
bokeh_plot.legend.border_line_color = color_honeydew

# Logo div sized to fill the bottom margin, right-aligned
logo_div = Div(
    text=f'''
        <div style="
            width: {bokeh_plot.width or 700}px;
            height: {LOGO_H + 40}px;
            position: relative;
            top: -{LOGO_H + 40}px;
            display: flex;
            align-items: center;
            justify-content: flex-end;
            padding-right: 40px;
            background-color: transparent;
            pointer-events: none;
        ">
            <img src="{logo_data_uri}" height="{LOGO_H}px"/>
        </div>
    '''
)

final = column(bokeh_plot, logo_div, spacing=0)
show(final)

The chart above is a good reminder that in investing, it’s sometimes beneficial to go against conventional wisdom. In their paper The 52-Week High and Momentum Investing, George and Hwang argued that investors often treat the 52-week high price as a psychological anchor. As a result, when a stock’s current price gets close to its 52-week high, investors hesitate to bid it higher, even when fundamentals suggest the price should continue rising. This behavior reflects underreaction because prices do not fully adjust to new positive information.

Because of this underreaction, stocks trading near their 52-week highs tend to continue rising afterward. This predictable pattern is known as the 52-week high effect. Put another way, prices near the 52-week high can signal momentum because investors are slow to push the stock toward its perceived true value.

This behavior is evident in the NVIDIA stock. Every time the closing price reaches the 52-week high, it may fall slightly, but it eventually climbs to an even higher 52-week high. What is especially surprising is that the 52-week high on January 5, 2024 is roughly the same as the 52-week low on January 2, 2025. Imagine being the investor on January 5, 2024 who believed the stock price couldn’t go any higher and chose to invest elsewhere. Oh, I can’t imagine the pain of losing such an opportunity!

However, it’s not always the case that a stock hitting its 52-week high will continue to rise. It’s important to pay attention to trading volume as well. When a stock approaches its 52-week high with high trading volume, it usually signals strong momentum. It’s like adding gasoline to a fire.

This momentum is often driven by genuine industry demand. Artificial intelligence (AI) is currently one of the most influential technologies in the world, and NVIDIA produces the chips that power much of this innovation. Unlike GameStop in 2021, whose rise was largely fueled by speculation and hype, NVIDIA’s growth is supported by strong demand for its products and the broader expansion of the AI industry.

Still, following the 52-week high is not a holy grail strategy. Instead, you should treat it as one tool in your broader investment toolbox.

The dataset

The dataset we’ll be analyzing contains crime data from neighborhoods in Los Angeles. I’ll read the dataset with pandas and store it in a variable called df. The Polars dataframe will be stored in a variable called data.

import pandas as pd

url = "https://data.lacity.org/api/views/2nrs-mtv8/rows.csv?accessType=DOWNLOAD"
df = pd.read_csv(url)
df.head()

I’ll read the dataset again with Polars.

import polars as pl

data = pl.read_csv(url)
data.head()

Selecting a row

The loc method in pandas can be used to select specific rows and columns. I’ll select the fifth row in the dataframe.

df.loc[5]

DR_NO                                                240412063
Date Rptd                               12/11/2024 12:00:00 AM
DATE OCC                                11/11/2020 12:00:00 AM
TIME OCC                                                  1210
AREA                                                         4
AREA NAME                                           Hollenbeck
Rpt Dist No                                                429
Part 1-2                                                     2
Crm Cd                                                     354
Crm Cd Desc                                  THEFT OF IDENTITY
Mocodes                                                   0100
Vict Age                                                    35
Vict Sex                                                     M
Vict Descent                                                 B
Premis Cd                                                502.0
Premis Desc       MULTI-UNIT DWELLING (APARTMENT, DUPLEX, ETC)
Weapon Used Cd                                             NaN
Weapon Desc                                                NaN
Status                                                      IC
Status Desc                                        Invest Cont
Crm Cd 1                                                 354.0
Crm Cd 2                                                   NaN
Crm Cd 3                                                   NaN
Crm Cd 4                                                   NaN
LOCATION               5300    CRONUS                       ST
Cross Street                                               NaN
LAT                                                     34.083
LON                                                  -118.1678
Name: 5, dtype: object

Let’s do the same with Polars. I’ll show two ways to select a specific row.

Version 1

(data
 .slice(5, 1)
 )

Version 2

(data
 .gather(5)
 )

Personally, I prefer version 2, but you can use whichever one you like.

If we wanted Polars to produce output closer to the pandas output, we could use to_dict to produce a list and then retrieve the first item in that list, which happens to be the dictionary we want.

(data
 .gather(5)
 .to_dicts()[0]
 )

{'DR_NO': 240412063,
 'Date Rptd': '12/11/2024 12:00:00 AM',
 'DATE OCC': '11/11/2020 12:00:00 AM',
 'TIME OCC': 1210,
 'AREA': 4,
 'AREA NAME': 'Hollenbeck',
 'Rpt Dist No': 429,
 'Part 1-2': 2,
 'Crm Cd': 354,
 'Crm Cd Desc': 'THEFT OF IDENTITY',
 'Mocodes': '0100',
 'Vict Age': 35,
 'Vict Sex': 'M',
 'Vict Descent': 'B',
 'Premis Cd': 502,
 'Premis Desc': 'MULTI-UNIT DWELLING (APARTMENT, DUPLEX, ETC)',
 'Weapon Used Cd': None,
 'Weapon Desc': None,
 'Status': 'IC',
 'Status Desc': 'Invest Cont',
 'Crm Cd 1': 354,
 'Crm Cd 2': None,
 'Crm Cd 3': None,
 'Crm Cd 4': None,
 'LOCATION': '5300    CRONUS                       ST',
 'Cross Street': None,
 'LAT': 34.083,
 'LON': -118.1678}

Although this is a dictionary, it is displayed vertically and therefore looks closer to the pandas output.

Selecting columns

Now let’s use loc to select a single column in df. The : means “return all the rows in the dataframe.”

df.loc[:, 'Crm Cd 1']

0          354.0
1          230.0
2          354.0
3          331.0
4          420.0
           ...  
1004889    510.0
1004890    745.0
1004891    341.0
1004892    230.0
1004893    510.0
Name: Crm Cd 1, Length: 1004894, dtype: float64

In Polars, we can display a single column by using the select method.

data.select('Crm Cd 1')

The first thing to note between these two outputs is that when a single column is selected in pandas, the output is a series, unlike in Polars, where the output is always a dataframe regardless of how many columns you select. Another difference is that the pandas output shows an index. Polars dataframes, on the other hand, do not have an index unless you create one.

If we really want the Polars output to be a series as well, we can do it with to_series.

(data
 .select('Crm Cd 1')
 .to_series()
 )

The two Polars outputs look the same, but they are different types. The way I distinguish between them is by looking at their shape at the top. The dataframe shows both the number of rows and columns: (1004894, 1), while the series only shows the number of rows: (1004894,).

Here’s how we can select two columns and all the rows in the pandas dataframe.

df.loc[:, ['AREA', 'TIME OCC']]

And here’s how to do it the Polars way.

(data
 .select('AREA','TIME OCC')
 )

Combine with list of rows and column labels

We can use a list to select the rows we want to return in both pandas and Polars dataframes.

df.loc[[0,1,2], ['AREA', 'TIME OCC']]

In a Polars dataframe, it is not necessary to use a list to select columns.

(data
 .gather([0,1,2])
 .select('AREA', 'TIME OCC')
 )

Selecting rows and columns over a range

Instead of listing the rows or columns we want to select, we can select them over a range. Let’s get the rows from 0 to 2 and the columns from TIME OCC to Part 1-2 in both dataframes.

df.loc[0:2, 'TIME OCC':'Part 1-2']

It turns out that in Polars, there is no direct way to select columns over a range by their names. However, we can achieve the same result as in the pandas example using two workarounds. Both require selecting columns by their index positions.

I’ll begin with the one that uses a list comprehension.

cols = data.columns
data.slice(0, 3).select(cols[cols.index("TIME OCC") : cols.index("Part 1-2") + 1])

I would argue that the above method is not idiomatic. List comprehensions are not native to Polars; remember that the library is written in Rust. Here’s the idiomatic approach.

(data
 .select(pl.nth(range(data.get_column_index('TIME OCC'), data.get_column_index('Part 1-2') + 1)))
 )

Because loc accepts list comprehensions, we can add logic to customize our row or column selection. The example below pulls out every third row and selects only the columns that contain ‘Crm’ in their names.

df.loc[[True if i%3 == 2 else False for i in range(len(df))],
       [True if 'Crm' in col else False for col in df.columns]] 

We can achieve similar results in Polars by creating an index or by using gather_every. The example below creates an index on the dataframe and performs a modulo calculation on the generated index numbers to get the desired rows. Then we use a regex pattern to select the columns that contain the string ‘Crm’.

(data
 .with_row_index()
 .filter(pl.col('index') % 3 == 2)
 .select(pl.col('^.*Crm.*$'))
)

An even more idiomatic way to write the above code is by using gather_every.

(data
1 .gather_every(3, offset=2)
 .select(pl.col('^.*Crm.*$'))
)

Just to show off how versatile Polars is, I’ll demonstrate other interesting ways to write the code above. I’ll begin with the one that uses the arange function. This also eliminates the need for an index.

(data
 .filter((pl.arange(0, pl.len()) % 3) == 2)
 .select(pl.col('^.*Crm.*$'))
 )

We can also use the int_range function to get the same result.

(data
 .filter((pl.int_range(pl.len()) % 3) == 2)
 .select(pl.col('^.*Crm.*$'))
)

Regex patterns are notorious for being difficult to remember. We can avoid them altogether by using special selectors in Polars.

import polars.selectors as cs

(data
 .filter(pl.int_range(pl.len()).mod(3) == 2)
 .select(cs.contains('Crm'))
 )

Boolean arrays

Most row filtering in pandas uses boolean masks. Let’s begin with a simple example: filter for rows where AREA NAME is ‘Central’. The code below produces a boolean mask.

df['AREA NAME'] == 'Central'

0          False
1          False
2          False
3          False
4          False
           ...  
1004889    False
1004890     True
1004891    False
1004892    False
1004893    False
Name: AREA NAME, Length: 1004894, dtype: bool

We can use this boolean mask in loc to get the desired rows in the dataframe.

df.loc[df['AREA NAME'] == 'Central']

By contrast, Polars does not require a boolean mask to perform this filtering operation.

(data
 .filter(pl.col('AREA NAME') == 'Central')
 )

Now let’s move to a more interesting example. The goal is to select all the columns that have more than the median number of unique values. We’ll calculate the number of unique entries in each column and then find the median value for those counts.

Let’s show the number of unique values in each column of the pandas dataframe.

df.nunique()

DR_NO             1004894
Date Rptd            1870
DATE OCC             1826
TIME OCC             1439
AREA                   21
AREA NAME              21
Rpt Dist No          1210
Part 1-2                2
Crm Cd                140
Crm Cd Desc           140
Mocodes            310912
Vict Age              104
Vict Sex                5
Vict Descent           20
Premis Cd             314
Premis Desc           306
Weapon Used Cd         79
Weapon Desc            79
Status                  6
Status Desc             6
Crm Cd 1              142
Crm Cd 2              126
Crm Cd 3               38
Crm Cd 4                6
LOCATION            66566
Cross Street        10413
LAT                  5426
LON                  4982
dtype: int64

Because the output is displayed vertically, calculating the median is as simple as calling median.

df.nunique().median()

np.float64(140.0)

The calculation is somewhat more involved in Polars. To begin with, the result showing the number of unique values in each column is displayed horizontally as a dataframe.

data.select(pl.all().n_unique())

We can calculate the median value 140.0 by using the transpose method or by using the concat_list function.

(data
 .select(pl.all().n_unique())
 .transpose()
 .median()
 .item()
 )

140.0

Transposing flips the dataframe from horizontal to vertical, making it possible to perform the median calculation.

The second approach collects the unique counts for each column into a list and then performs the median calculation.

(data
 .select(pl.all().n_unique())                   
 .select(pl.concat_list(pl.all()).list.median())
 .item()                                        
)

140.0

Now that we know the median value is 140.0, we can use it to get the desired columns. Here’s how to do it in pandas.

df.loc[:, df.nunique() > 140]

And here’s how to do it in Polars.

(data
 .select(col for col in data.columns if data.select(pl.col(col).n_unique()).item() > 140)
 )

Closing with a practical example

Let’s end with a visualization of the previous analysis. I’ll focus on the Burglary from Vehicle column and plot it over time for the different AREA NAME values. Before we begin creating the charts, I’ll show the dataframe that will be used to create the plot so we know what it looks like.

Here’s the pandas version.

(df
 .astype({'Date Rptd': 'datetime64[ns]',
          'DATE OCC': 'datetime64[ns]',
          })
 .groupby([pd.Grouper(key='DATE OCC', freq='ME'), 'AREA NAME', 'Crm Cd Desc'])
 .size()
 .unstack()
 .loc[:, 'BURGLARY FROM VEHICLE']
 .unstack()
1 .loc[:, lambda adf: sorted(adf.columns, key=lambda col: 1 if col == 'Hollywood' else -1)]
)

And here’s the Polars version, which I’ll store in a variable called to_plot to use later.

to_plot = (data
 .with_columns(pl.col('Date Rptd', 'DATE OCC').str.strptime(pl.Datetime, '%m/%d/%Y %I:%M:%S %p'))
 .sort('DATE OCC')
 .group_by_dynamic('DATE OCC', every='1mo', group_by=['AREA NAME', 'Crm Cd Desc'])
 .agg(pl.len())
 .filter(pl.col('Crm Cd Desc') =='BURGLARY FROM VEHICLE')
 .select(pl.exclude('Crm Cd Desc'))
 .pivot(on='AREA NAME', index='DATE OCC', values='len')
1 .select(pl.all().exclude('Hollywood'), 'Hollywood')
)
to_plot

Creating the charts

With pandas’ built-in plot

It turns out that pandas has built-in plotting capabilities, so we do not have to import a charting library.

def set_colors(df):
    global colors
    colors = ['#991111' if col == 'Hollywood' else '#999999' for col in df.columns]
    return df

ax = (df
 .astype({'Date Rptd': 'datetime64[ns]',
          'DATE OCC': 'datetime64[ns]',
          })
 .groupby([pd.Grouper(key='DATE OCC', freq='ME'), 'AREA NAME', 'Crm Cd Desc'])
 .size()
 .unstack()
 .loc[:, 'BURGLARY FROM VEHICLE']
 .unstack()
 .loc[:, lambda adf: sorted(adf.columns, key=lambda col: 1 if col == 'Hollywood' else -1)]
 .pipe(set_colors)
 .plot(color=colors, title='BURGLARY FROM VEHICLE in LA', figsize=(10,6))
)
ax.legend(bbox_to_anchor=(1,1), ncol=2);

With hvplot

HvPlot is one of the few charting libraries that does not require converting the data into a pandas dataframe or a Python list to create charts.

import hvplot.polars

(to_plot
 .pipe(lambda df_: df_.hvplot.line(
       x='DATE OCC',
       y=(cols := [c for c in df_.columns[1:] if c != 'Hollywood'] + ['Hollywood']),
       color=['#999999'] * (len(cols) - 1) + ['#991111'],
       title='BURGLARY FROM VEHICLE in LA',
       legend_opts={'title':'Neighborhood'},
       legend='top_left', legend_cols=3, height=575, width=900))
)

With glyphx

I’ve been experimenting with the Glyphx charting library after hearing about it from Christopher Bailey on The Real Python Podcast. It captured my interest because the charts created with this library are in SVG format. This is handy because you can export the chart and enhance it in a tool like Inkscape.

However, Glyphx currently does not support Polars dataframes. We have to pass a list to the x-axis or y-axis to create the chart.

from glyphx import Figure
from glyphx.series import LineSeries

non_hollywood = [col for col in to_plot.columns[1:] if col != 'Hollywood']

dates = to_plot['DATE OCC'].to_list()

fig = Figure(width=800, height=450).set_title('BURGLARY FROM VEHICLE in LA')

for col in non_hollywood:
    fig.add(LineSeries(dates, to_plot[col].to_list(), color='#bbbbbb', label=col))

fig.add(LineSeries(dates, to_plot['Hollywood'].to_list(), color='red', label='Hollywood'))

fig.set_legend('right').tight_layout()
fig.show();

Other limitations I encountered are:

1. The circle markers on the lines cannot be removed. Because the data points are very close to each other in the lower part of the chart, the markers make the chart look more like a scatterplot than a line chart.
2. It’s not possible to display the legend in a three-column layout like in the HvPlot chart. I do not like the long list of neighborhoods, so I ended up listing only the neighborhood of interest in the legend.

from glyphx import Figure
from glyphx.series import LineSeries

non_hollywood = [col for col in to_plot.columns[1:] if col != 'Hollywood']

dates = to_plot['DATE OCC'].to_list()

fig = Figure(width=800, height=450).set_title('BURGLARY FROM VEHICLE in LA')

for col in non_hollywood:
    fig.add(LineSeries(dates, to_plot[col].to_list(), color='#bbbbbb'))

fig.add(LineSeries(dates, to_plot['Hollywood'].to_list(), color='red', label='Hollywood'))

fig.set_legend('right').tight_layout()
fig.show();