The number can be anything, is there a number at which "we don't know" starts?
The model's parameters are in your RAM, you insert the prompt, it runs through the model and gives you a result. I'm sure if you spend a bit of time, you could add some software scaffolding around the process to show you each step of the way. How is this different from a statistical model where you "do know"?
For just a few parameters, you can understand the model, because you can hold it in your mind. But for machine learning models that's not possible, as they are far more complex.
The model's parameters are in your RAM, you insert the prompt, it runs through the model and gives you a result. I'm sure if you spend a bit of time, you could add some software scaffolding around the process to show you each step of the way. How is this different from a statistical model where you "do know"?