Skip to content

Commit

Permalink
Fixes.
Browse files Browse the repository at this point in the history
  • Loading branch information
eaplatanios committed Jan 22, 2024
1 parent 67fe59c commit 41b3f5a
Show file tree
Hide file tree
Showing 2 changed files with 25 additions and 0 deletions.
8 changes: 8 additions & 0 deletions tokenizers/src/decoders/sequence.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@ impl Sequence {
pub fn new(decoders: Vec<DecoderWrapper>) -> Self {
Self { decoders }
}

pub fn get_decoders(&self) -> &[DecoderWrapper] {
&self.decoders
}

pub fn get_decoders_mut(&mut self) -> &mut [DecoderWrapper] {
&mut self.decoders
}
}

impl Decoder for Sequence {
Expand Down
17 changes: 17 additions & 0 deletions tokenizers/src/tokenizer/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,12 @@ where
self
}

/// Set the added vocabulary.
pub fn with_added_vocabulary(mut self, added_vocabulary: AddedVocabulary) -> Self {

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.9)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.8)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.10)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.7)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 388 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `tokenizer::TokenizerBuilder::<M, N, PT, PP, D>::with_added_vocabulary`
self.added_vocabulary = added_vocabulary;
self
}

/// Set the trunaction parameters.
#[must_use]
pub fn with_truncation(mut self, trunc: Option<TruncationParams>) -> Self {
Expand Down Expand Up @@ -598,6 +604,17 @@ where
&self.model
}

/// Set the added vocabulary.
pub fn with_added_vocabulary(&mut self, added_vocabulary: AddedVocabulary) -> &mut Self {

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.9)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.8)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.10)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.7)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`

Check warning on line 608 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::with_added_vocabulary`
self.added_vocabulary = added_vocabulary.into();
self
}

/// Get the added vocabulary
pub fn get_added_vocabulary(&self) -> &AddedVocabulary {

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.9)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.8)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.10)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check it builds for Windows 32-bit (3.7)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (ubuntu-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`

Check warning on line 614 in tokenizers/src/tokenizer/mod.rs

View workflow job for this annotation

GitHub Actions / Check everything builds & tests (macos-latest)

type `added_vocabulary::AddedVocabulary` is more private than the item `TokenizerImpl::<M, N, PT, PP, D>::get_added_vocabulary`
&self.added_vocabulary
}

/// Set the truncation parameters
///
/// Fails if `stride` is too high relative to `max_length` and `post_processor.added_tokens()`
Expand Down

0 comments on commit 41b3f5a

Please sign in to comment.