IPFS also has issues with large datasets, anything larger then a few TBs cases nodes to explode and indexing takes forever.

Also the lookup time for files is too slow to use it on anything user facing.

Start clicking on stuff, You'll find it :)

I was clicking around like mad, nothing happened and I gave up

That whole list is a hoot,

has_toxicity_score_above_threshold

is a interesting value, I wonder were the 0.91 was though up at

I avoid Javascript outright because async/await/promise is confusing to me. I blame it on being a PHP Programmer and likes things to run serially.

I felt the same way coming from a threaded language.

Learning the event loop, then promises, then async/await is a must. Today, you probably should throw typescript on top.

A steep learning curve just to get back to a typed language that can do things concurrently.

You do get used to it, but it is a mess of stuff.

Threads are their own steep learning curve, I think it's just hard to do two things at once.

It's easy to do two things at once when you can ask two different entities to do them for you (threads).

What's hard is thinking about how to coordinate the work they are doing for you: when to consider them done, how to ask them if they did the work successfully, what to do if they need to use the same tool at some point during the work etc.

This is ridiculous. Handling real threads is much more complicated than handling async calls and the event loop of JavaScript.

Languages with threading require learning techniques to use them safely and many, including myself, have learned how.

Even if concurrency is easier to get right on node I'd say the node ecosystem has just layered on complexity in other ways to get to something just as difficult to use overall.

Promises and async/await sugar are only the tip of the iceberg.

/r/gatekeeping

It drove me crazy too, until I needed to use Puppeteer which requires you to write async/await (there are Puppeteer implementations in other languages, but they all seem to make compromises I didn't want). Generally speaking, async/await allows you to write code that looks and feels serial. Perhaps try using one of the async libraries for PHP to wrap your mind around the concept of async/await (like https://github.com/spatie/async)

Hyperscript can help with this. https://hyperscript.org/

Makes using a bit of JavaScript relatively simple, just not much in Stack Exchange yet which means reading docs..

The biggest feature of PHP for new people to the language is in-lining

You can do stupid stuff like

<table> <?php foreach($array as $row){ ?> <tr><td><?php echo $row[0];?></td></tr> <?php } ?>

Meaning you can just inlay PHP in your raw HTML as you please, this example will reprint the <tr> and <td> for every key in $array

I am working on a MicroPHP to shove into something like a RP2040 since PHP is really just a shell script for a whole bunch of C functions... but PHP eats too much ram currently on the micros

Two low end Hetzner/OVH Boxes for redundancy should do the trick

That's why AWS charges so much for outgoing traffic.

I wonder what mine will end up being

That's definitely a UFO

Yours is also a UFO.

IPFS is not for long term data storage, and once you get about the 100k file mark your tables are so damn slow you can't add anything new

Full Archival with the standards required by the Internet Archive require that full unmodified headers are required, and unmodified content. This tends not to work well with modern browsers. Chrome and Firefox both fail at this currently. Someone is looking into a kind of modified Firefox to help with this. but its just not that how this system works. Now the Archive.org does have a API of sorts to say hay archive this URL, and a little working on the backend goes and does it..

What the Archive Team does is on a much more massive scale. Like SETI at home scale of scraping data across the internet. At almost every point we have had to make custom tools to ensure it meets our needs in our archival efforts.

> standards required by the Internet Archive require that full unmodified headers are required

Sure, this would not be a solution for the Wayback Machine, but would be adequate[1][2] for lots of non-Wayback collections (of the sort that Archive Team is associated with).

1. https://twitter.com/textfiles/status/970912494284779520

2. http://ascii.textfiles.com/archives/4285