Merging Embeddings with Temporal Data

Merging Embedding Data

Motivation: Static meta-data has the ability to potentially improve temporal forecasts. In many cases meta-data is high dimensional. Therefore meta-data must be separately embedding into a vector before being incorporated.

Goals: Enable Flow-Forecast users to effectively leverage temporal meta-data in their forecasts and classification problems.

For information on how this relates to COVID-19 forecasts see here

Core Architectural Solution

Continue training meta-embedding model

Model performance benefits:

• Embedding model would likely produce more task specific embedding that incorporate the temporal nature.
• Performance would (likely) improve more with training the model.

Technical Implementation

• We would have to initialize two models at once in one form or another. (1) The time series forecast models and (2) The model embedding model. Depending on how this works we might have to pass meta-model parameters directly to the time series PyTorch module.

• We would either need to extend one of the data loaders to handle returning the meta-data or we would need to leverage both data loaders with some primary key to keep both in sync.

Key questions

Would the meta_embedding model have to initialized in the actual torch.nn.Module?

Save static embedding to cloud storage and enable retrieval from there

Combining of Meta-Embedding Data

Vanilla Concatenation

Bi-linear layer

Bi-linear pooling

Other

Testing Notebooks:

We have created the following notebooks for testing out concatenation techniques.