Linklog

An interesting perspective on the abstraction behind the dataframe algebra.

This article highlights a very interesting aspect of analysing historical data. In this case, center penalty kicks appear surprisingly successful. What this article shows is that one shouldn't blindly assume that it's because they are inherently more effective (they aren't), it's because goalkeepers act suboptimally and the player-goalkeeper interaction is responsible for this overrepresentation.

If like me you feel late to the dbt party, we're not alone!

A fun analysis of the birthday effect using actual data and thorough methodology.

I've been working a lot on our data pipelines at work, switching to polars mostly for performance and introducing rigorous checks and validations of data at various stages. I haven't yet used dataframely, but its principle really resonates with my use case, so I recommend checking it out.

Succinct data structures are clever ways to pack a lot of information in lightweight structures like bit vectors. A very interesting read!

A description of the effect of binary quantization on embeddings. By restricting the dtype of embedding vectors, you can get a tradeoff between accuracy in latent space and size of the embedding. Using binary dtype seems to conserve a surprisingly high amount of the original information content (about 97%) while yielding a gigantic amount of saving in space (about 97% too here).

Musings about the use of data contracts to validate data sources. If you've ever been frustrated by a data source suddenly changing its schema or sending unexpected data, this is for you!

Article about the use of Polars for both production and development stages. When starting with Polars, I found it easy to write production code (usually a long pipeline of LazyFrames ending with a collect), but struggled with writing optimal development code.

Great online book about Polars targeted to Pandas users. If you haven't heard about Polars yet, do yourself a favor and read this.

A collection of resources to help make better data visualizations. Definitely useful as a refresher or reference before making a report or a presentation.

I always love when data scientists take it too far on their hobbies. This is a cool example of data science applied to "The Office", to figure out through sentiment analysis if Michael Scott was truly appreciated.

An interesting explanation of implicit data contracts through naming conventions.