Skip to content

Commit

Permalink
Merge branch 'add-display' of github.com:huggingface/tokenizers into …
Browse files Browse the repository at this point in the history
…add-display
  • Loading branch information
ArthurZucker committed Jun 8, 2024
2 parents df51116 + 35373de commit 4b4b833
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions tokenizers/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
//! 4. The `PostProcessor`: in charge of post-processing the `Encoding` to add anything relevant
//! that, for example, a language model would need, such as special tokens.
//!
//! ## Loading a pretrained tokenizer from the Hub
//! ## Loading a pretrained tokenizer from the Hub.
//! ```
//! use tokenizers::tokenizer::{Result, Tokenizer};
//!
Expand All @@ -39,7 +39,7 @@
//! }
//! ```
//!
//! ## Deserialization and tokenization example
//! ## Deserialization and tokenization example.
//!
//! ```no_run
//! use tokenizers::tokenizer::{Result, Tokenizer, EncodeInput};
Expand Down

0 comments on commit 4b4b833

Please sign in to comment.