Chunks in Data Drift and regression tasks

[lolloloschi97]

Hi, thanks for the library it's an exciting topic! I've two questions:

1. Is it possible to use the library even for regression tasks? I've tried to initialise binary classification tasks with fictional probability and target columns to use the Data Drift algorithm, but still, I've no idea how to compute performance estimation.

2. Univariate and Multivariate Data Drift works with chunks, which if I've understood them well are consecutive. Sometimes in time series, we might want to compare a chunk wrt analogous period in the previous years, is there a way to define this comparison?

Thanks in advance, Lorenzo.

[SoyGema]

Hello there Lorenzo,
Gema's here. Nice to e-meet you. I've been using the beta ( mostly v0.3.2 and now v0.4.0) my answer comes mostly from my experimentation with v0.3.2) 👍
As far as I know :

1. Regression is in the roadmap but not yet implemented
2. This is what I know about chunking . Chunks are created to estimate metrics and they are key for both estimating post-deployment performance ( CBPE ) and calculate Data Drift (ej : UnivariateStatisticalDriftCalculator class ) . As far as I'm understanding, you might be interested in time-based chunking, which creates chunks based on time periods . You can read about it more here - please , don't hesitate to correct me If I didn't understand your question well -

Hope this helps! thanks for opening up a discussion . Happy to dig with you with this

[nnansters]

Hey Lorenzo,

sorry for not getting back to you earlier, but as you might have noticed in our latest announcement we've just added something of relevance to your question!

We've just added some very basic functionality for regression use cases. You can now create / extract metadata for regression cases and use that metadata to detect data drift.
We're still performing research on performance estimation and calculation, so more news to follow there.

Concerning the second part of your question: as @SoyGema mentioned the time-based chunking should give you what you need here provided you have enough data to cover a timespan of multiple years.
You can chunk on a period of days, months or even years. The drift calculation results will then give you some metrics for each chunk. You should then be able to select the chunks corresponding to the same day or month for the different years.

The explanation is somewhat abstract, feel free to ask me for any clarification!

[nnansters]

Hey @YYYasin19 , thanks for trying out our library!

Good question, the principle of what we use reference data for might not be very clear.
We currently use it to train or fit some things like imputers, scalers, PCA or whatever the calculator uses internally. Reference data is being used as a training set here.
More important: we also use it to calculate some thresholds, e.g. by using the calculation results on the reference data, taking the mean and adding/subtracting 3 times the standard deviation. This results in fixed upper and lower thresholds that are then stored as internal state of the calculator.

When performing calculations on new data we're not really comparing to reference directly, we just check if the calculated results are exceeding these fixed thresholds. So: any comparison with the reference data is strictly aggregated.

Comparing on a month to month basis is something we currently don't do but most of the calculators store the results of the calculations on reference data as a property called previous_reference_results (version 0.5.0 and up). We use it for visualization purposes. But as a workaround for now you could get that property value and use it for manual comparison. Used the PeriodBasedChunker should make that very easy.

What kind of calculation are you performing?

Sorry for the wall of text without examples, currently on holiday and replying on my phone 📱

[YYYasin19]

Hey,

thanks for the super-fast and in-depth reply.

But would the current procedure not skew my results?
A PCA trained on the whole reference dataset might give me a different result than a PCA just trained on all previous months over the years, right? calculate(reference, chunk[period]) vs. calculate(reference[period], chunk[period])
I have looked into the previous_reference_results attribute and each row i seems to keep the results of comparing reference_df and chunk[i].
Maybe there's just a misunderstanding in methodology/implementation, though, I will look further into it.

Hope you're able to enjoy your holiday!
Best, Yasin

[nnansters]

Hey Yasin, the reference results contains the result of the following process:

• call the _calculate function on reference data
• during this calculation the reference data gets chunked and the result will contain the calculated/estimated metric for each chunk of reference data

The reference results are chunked in the same way as your other data later, so you can compare the metric values on reference chunks vs. those on 'analysis' chunks. Hope that makes sense?

As for your first remark: yes indeed, we do consider the full reference period, since we want to capture temporal patterns as well.
Not entirely sure if it makes sense to have other options here, paging dr. @nikml for an expert consult here!

[nikml]

Hello @YYYasin19

As @nnansters has said, we use the whole reference dataset to perform drift comparisons on the chunks. The periodic use case you referred to on this post is not covered. You are right that our approach does not account for periodic effects.

If you know the period of such effects in your data you can select your reference data to only come from that period. Then you should see no drift on times where you expect the effect to come back. It's not a full solution to the issue you have, but unfortunately, identifying periodic effects within the data is not something we have worked on and can account for at the moment.

If you can think of better ways that we could address this issue please say. We may not get back to it immediately but we want to be able to account for periodic effects in data at the future.

[YYYasin19]

Hey,

thanks again you both; your solutions and comments are very valid and I have looked into them.
Indeed, just selecting the correct periods for my reference_data seems to be a good trick to solve that case for now.

My solution (and I could draft this in a PR if you think it fits on your roadmap) would be to filter the reference dataset within the chunk loop (pseudo-code):

for chunk in chunks:
  
  if isinstance(self.chunker, PeriodBasedChunker) and self.compare_corresponding_periods:
    corr_period_selector = self.previous_reference_data[column].dt.week == chunk.week # one example
    reference_column = self.previous_reference_data.loc[corr_period_selector, [column]]
    
  # ...
  # regular data check

The advantage (compared to just passing in the filtered reference_df from the user side) is that the plots are now "full" plots and don't look filtered.