byte-level language modeling megabyte vs. transformer

language models don't read words like humans do. they break down each individual words, into hundreds of thousands of tiny pieces, then organizes them into a dictionary of vocabulary to be used in a language model. the MEGABYTE architecture explored the idea of what if the model looks at one byte at a time, instead of tokenizing each word. as part of my NLP course's emperical final project, me and abigail lee has attempted to recreate yu et. al's paper on an experimental transformer architecture.