How big are our embeddings now and why?
Embedding sizes and architectures have changed remarkably over the past 5 years
Summary
Embeddings are numerical representations that map words, sentences, or images into a continuous vector space to facilitate machine processing and understanding.
Token embeddings in models like GPT-3 represent semantic and syntactic properties and are refined through multiple layers, with each token occupying a high-dimensional space. Techniques for generating embeddings have evolved from simple methods like one-hot encoding and TF-IDF to complex models such as Word2Vec, GloVe, FastText, and transformers, enabling advanced contextual understanding in AI.