I am interested in making a system that can memoize large databases from ETL systems and then serve that on iroh or ipfs/torrent, such that a process that may take a supercomputer a week to process can have the same code run on a laptop and it will notice it's been done my a university supercomputer before already and grab that result automatically from the decentralized network of all people using the software (who downloaded the ETL database).
Derivations are just a set of instructions combined with a set of inputs, and a unique hash is made from that.
If you make a derivation whose result is the invocation of another, and you try and grab the outcome from that derivation, here’s what will happen:
- it will generate the hash
- it will look that hash up in your local /nix/store
- if not found it will look that hash up in any remote caches you have configured
- if not found it will create it using the inputs and instructions
This is transitive so any missing inputs will also be searched for and built if missing, etc.
So if the outcome from your process is something you want to keep and make accessible to other machines, you can do that.
If the machines differ in architecture, the “inputs” might differ between machines (e.g. clang on Mac silicon is not the same as clang on x86-64) and that would result in a different final hash, thus one computation per unique architecture.
This is ultimately the correct behaviour as guaranteeing identical output on different architectures is somewhat unrealistic.
I see. Perhaps the added benefit I am trying to create with this other system is that specifying remote locations isn't necessary, and is just inherited as the distributed network. Anytime anyone runs it, they're added to the network, so it scales with the number of users.
I am interested in making a system that can memoize large databases from ETL systems and then serve that on iroh or ipfs/torrent, such that a process that may take a supercomputer a week to process can have the same code run on a laptop and it will notice it's been done my a university supercomputer before already and grab that result automatically from the decentralized network of all people using the software (who downloaded the ETL database).
That way you save compute and time.