polars quantile example

[mccalluc]

• Fix Example of polars quantile workflow #1500
• Tweaked column names and presentation order for clarity
• Just playing with the numbers, the behavior of quantile confused me, but it doesn't need to block this.
• We should also say where scale, alpha, and param are coming from.

[raprasad]

More context for the twice-key and ones columns may be needed to make the example less abstract; possibly including changing the names

[mccalluc]

By "less abstract", do you mean the dataset should actually look like an interesting, real-world dataset? I think examples like that can be useful, but if we're just trying to explain the machinery, constants etc. can make it easier to see what the DP adds.

I'll expand the explanation and maybe we can talk on Thursday.

[Shoeboxam]

I like the thought of using a real-world data set, even if it's just PUMS. This could be the basis of the long-awaited cookbook docs page.

[mccalluc]

Going to move back to draft to add more context.

[mccalluc]

Going to take a break from this, and pick it up when alias is fixed upstream.

[mccalluc]

I'm still a bit confused by the behavior.

some data:

>>> private_lf = pl.LazyFrame([
...     pl.Series("grouping-key", [1, 2, 3, 4, 5] * 10, dtype=pl.Int32),
...     pl.Series("noisy-key", [1] * 10 + [1, 2, 3, 4, 5] * 6 + [5] * 10, dtype=pl.Int32),
...     pl.Series("ones", [1.0] * 50, dtype=pl.Float64),
... ])

a plan:

>>> quantiles_plan = empty_lf.group_by("grouping-key").agg([
...     pl.col("noisy-key")
...         .dp.mean(bounds=(1, 5), scale=2.)
...         .alias("mean"),
...     pl.col("noisy-key")
...         .dp.median(candidates=[1, 2, 3, 4, 5], scale=1.0)
...         .alias("median"),
...     pl.col("noisy-key")
...         .dp.median(candidates=[1, 3, 5], scale=1.0)
...         .alias("median 1/3/5"),
...     pl.col("noisy-key")
...         .dp.quantile(candidates=[1, 2, 3, 4, 5], alpha=0.1, scale=1.0)
...         .alias("10% quantile"),
...     pl.col("noisy-key")
...         .dp.quantile(candidates=[1, 2, 3, 4, 5], alpha=0.9, scale=1.0)
...         .alias("90% quantile"),
... ])

the result:

>>> print(quantiles_release) # doctest: +ELLIPSIS
shape: (5, 6)
┌──────────────┬──────┬────────┬──────────────┬──────────────┬──────────────┐
│ grouping-key ┆ mean ┆ median ┆ median 1/3/5 ┆ 10% quantile ┆ 90% quantile │
│ ---          ┆ ---  ┆ ---    ┆ ---          ┆ ---          ┆ ---          │
│ i32          ┆ f64  ┆ i64    ┆ i64          ┆ i64          ┆ i64          │
╞══════════════╪══════╪════════╪══════════════╪══════════════╪══════════════╡
│ 1            ┆ ...  ┆ 5      ┆ 5            ┆ 5            ┆ 1            │
│ 2            ┆ ...  ┆ 5      ┆ 5            ┆ 5            ┆ 1            │
│ 3            ┆ ...  ┆ 5      ┆ 5            ┆ 5            ┆ 1            │
│ 4            ┆ ...  ┆ 5      ┆ 5            ┆ 5            ┆ 1            │
│ 5            ┆ ...  ┆ 1      ┆ 1            ┆ 5            ┆ 1            │
└──────────────┴──────┴────────┴──────────────┴──────────────┴──────────────┘

• Am I misusing median?
• alpha works opposite from what I was expecting, but my expectations are likely wrong.

[Shoeboxam]

I like the thought of using a real-world data set, even if it's just PUMS. This could be the basis of the long-awaited cookbook docs page.

[Shoeboxam]

Separate docs page would be useful for explaining the descriptors on frame domains.

[mccalluc]

Supplanted by Gurman's work. Closing.