Implemented option to replace embedding vectors by non-trainable 1-hot encoding vectors #48
Conversation
TimotheeMickus
requested review from
TimotheeMickus
and removed request for
TimotheeMickus
mammoth/modules/embeddings.py
Outdated
| embeddings = [nn.Embedding(vocab, dim, padding_idx=pad) for vocab, dim, pad in emb_params] | ||
|
|
||
| else: | ||
| print("CREATING EMBEDDINGLESS") |
There was a problem hiding this comment.
being pedantic here: would like to see a more informative log output (IFF not only for debugging purposes).
perhaps print using the logger.
There was a problem hiding this comment.
Fixed this. Was there for debugging purposes.
| @@ -32,8 +32,9 @@ def attention_bridge_optimizer(model, task_queue_manager, base_optimizer): | |||
| params.append(param) | |||
| if name in suboptimizers: | |||
| raise Exception(f'Trying to create second optimizer for "{name}"') | |||
| optimizer = base_optimizer(params) | |||
| suboptimizers[name] = optimizer | |||
| if len(params) != 0: | |||
There was a problem hiding this comment.
just out of curiosity: why can params now be empty? i am not sure how this optimizer could interact only with the embeddings
There was a problem hiding this comment.
This fixes a problem in which a sub-optimiser is created for the embedding layer but there are no trainable parameters there since the weights are frozen.
mammoth/opts.py
Outdated
| '--enable_embeddingless', | ||
| '-enable_embeddingless', | ||
| action='store_true', | ||
| help="Enable the use of embeddingless models.", |
There was a problem hiding this comment.
Perhaps add a reference here or a tiny description of what embeddingless is here -- e.g. byte level something something implementation based on citation_name et al. (2045)
No description provided.