For the sake of argument, I'd like us to consider current generative AI models as compressed forms of knowledge. Individual tokens can exist in multiple locations within this compressed knowledge. The token "ABC," for example, could live in the portion of the cloud that knows about Sesame Street, the alphabet, Jackson Five lyrics, etc. LLMs condense their training set into probabilistic domains. The number of parameters for a model roughly corresponds to the amount of knowledge compressed into that model. For example, you can get a 12 billion parameter version of Google's Gemma4 model in Ollama. This model is roughly 7.4 gigabytes in size. This is actually a pretty decent model, but when you look at this model, you see that 7.4 gigabytes is really small for what you can do with it
I am also going to recommend the book "Why Machines Learn: The Elegant Math Behind Modern AI" by Anil Ananthaswamy (sponsored link) for a more technical underpinning than I'm going to go into in this post.
The four main root causes that I propose are causing AI hallucinations are the input set, the compression mechanism, the precision of floating point models on computers, and the steps taken to quantize these models.
First, the input set. Current generative AI models predict what the next likely token is going to be in an output set based on the input that was trained into it. The "temperature" that you pass into a model will give lower likelihood tokens a higher chance of being chosen. A low temperature (e.g. less than 0.3) will result in the highest likelihood of an appropriate response, while a high temperature (e.g. greater than 0.8) will give lower likelihood tokens a chance at coming out. The Gemma4 model linked earlier has a default temperature of 1.0. Since some level of randomness is going to be coming back for any temperature greater than 0.0, the data that the model was trained on will have a massive impact on the potential output set for tokens at any one time. Models that are purpose trained (e.g. coding models) are more likely to have appropriate responses, but they're still going to be limited by the training data. If you train a coding model on nothing but the results of the final projects from a weekend game jam, chances are that the output won't necessarily be high quality. Most code that you find on the internet (mine included) will be of average quality at best, so it's no surprise that you'll find most generated code is middling-to-average at best. The more specific you are in your request, the more likely that the model will end up with higher quality output, but if any amount of "bad code" makes it into the model, there's a chance that "bad code" can still come out. If your input data contains any invalid or malicious input tokens, there's still a chance that can come out, and the more questionable data you train your model on, the more likely that you'll get either invalid data that matches your query or inappropriate data that reinforces bad ideas that you have.
Second, the compression mechanism. Let's try to look at a bit of an analogue for compression: images. When you are trying to get an image to take up less space on disk or in memory, there are usually only a couple of ways of doing it. You can reduce the resolution (e.g. reduce the resolution from 3840x2160 to 1280x720), or you can use a lossy compression algorithm (e.g. compress a RAW image as a JPEG). In the end, the image gets compressed heavily, but you lose fine detail. The same thing happens when you compress knowledge into a model. Google's Gemma3 model compressed 14 trillion input tokens into a 27 billion parameter model roughly 17GB in size. Assuming an average input token size of 3 bytes, that's still compressing the knowledge contained in those input tokens down to roughly 1% the size of the original input data. Take a selfie and compress it down to 1% the size of the original image. You'll still be able to tell that it's you, but a ton of information is lost.
While information was lost, you can still tell what the original source was. The training mechanism used to backpropagate the knowledge is the same. You are still working with the knowledge, but the fine details can get lost depending on how the training occurs.
Third, the precision of floating point models on computers. If you've ever worked on video games, you may have started seeing issues when game worlds got large. The further things got from origin or the larger they got, the more likely you were to start seeing precision issues. Fabian Sanglard did a great breakdown on the most common floating point format used on PCs. So why does this matter? The more knowledge you are encoding into a model, the more space you need to have between your knowledge clusters. The larger the numbers, the more likely that some of the finer points of data could get dropped out of the window. This won't matter much at this point given that this is all meant to define a hyperplane in n-dimensional space, but it will definitely start having an issue when we get to our final problem: quantization.
When models are stored in pure floating point land (FP32 or better), they can be very accurate, but memory hungry and slow. Most models that we use are quantized, meaning that they are brought down to a lower precision version. All quantized models are lossy, meaning you are losing some level of detail. Let's look at a common quantization format: NVFP4. If you go back to Fabian's post, rather than using 32 bits for a number, it is compressing down the entire floating point number to four bits with values [-6, -4, -3, -2, -1.5, -1.0, -0.5, -0.0, 0.0, 0.5, 1.0, 1.5, 2, 3, 4, 6]. For every group of sixteen numbers, there is an additional FP8 scalar saved as well which scales every number within that group either up or down. There is a lot of min-maxing used in the encoding process with losses measured using RMSE, but in the end, you're still losing some data. Even when you are able to minimize it down to 1-2%, it's already extra loss on top of a lossy compression algorithm. It's like saving a lossy JPEG at a lower compression level. These small errors continue to propagate through the model, through the prompt processing pipeline, though the token generation pipeline, etc. NVFP4 is currently among the best because it also includes an additional scaling factor per tensor, but as you experiment with lower and lower quantization models, you'll continue to find worse and worse results because just so much fine detail is getting lost. Most numbers in a model are going to be close to zero or close to the max, but "close to" and "at" are very different ideas. The quantization algorithm is there to try to minimize data loss, but it can't eliminate it.
So what does all this have to do with hallucinations? It's in the math. The input set is indiscriminate in what it encodes, so any stray token could end up in its prediction set, whether appropriate or not. The compressed knowledge set is lossy, so there's no guarantee that the proper context for that token is present or not. The temperature of the model could easily be set to where the bad token is included. The location of that bad token in our knowledge space could be near a precision boundary in its initial training runs. The quantized model that we are using could shift the token closer to the valid values making it more likely to be used when it shouldn't. Once the model starts going down an incorrect or hallucinating path, that's all there is to it.
(Note that this is a general hypothesis. I'm writing this to try to get my thoughts out, and I'm leaving this post marked as a draft as I may come back and re-edit based on additional learnings.)
