Genomic Tokenizer: Toward a biology-driven tokenization in transformer models for DNA sequences https://t.co/yN90IYcZA7 🧬🖥️🧪 https://t.co/uBG9ryWF8a

Genomic Tokenizer ensures biological nuances inherent in genetic variations preserved during tokenization. The vocabulary includes all possible codons, but synonymous codons coding for the same amino acids are assigned the same IDs, improving the efficiency of the tokenizer.

Genomic Tokenizer (GT) incorporates start codons, synonymous codons, and stop codons into a tokenizer interface of the HuggingFace transformer package, giving it the ability to handle shifts in reading frames caused by nucleotide additions or deletions within DNA sequences.