Lots of people make the mistake of thinking there's only two vectors you can go to improve performance, high or wide.
High - throw hardware at the problem, on a single machine
Wide - Add more machines
There's a third direction you can go, I call it "going deep". Today's programs run on software stacks so high and so abstract that we're just now getting around to redeveloping (again for like the 3rd or 4th time) software that performs about as well as software we had around in the 1990s and early 2000s.
Going deep means stripping away this nonsense and getting down closer to the metal, using smart algorithms, planning and working through a problem and seeing if you can size the solution to running on one machine as-is. Modern CPUs, memory and disk (especially SSDs) are unbelievably fast compared to what we had at the turn of the millenium, yet we treat them like they're spare capacity to soak up even lazier abstractions. We keep thinking that completing the task means successfully scaling out a complex network of compute nodes, but completing the task actually means processing the data and getting meaningful results in a reasonable amount of time.
This isn't really hard to do (but it can be tedious), and it doesn't mean writing system-level C or ASM code. Just seeing what you can do on a single medium-specc'd consumer machine first, then scaling up or out if you really need to. It turns out a great many problems really don't need scalable compute clusters. And in fact, the time you'd spend setting that up, and building the coordinating code (which introduces yet more layers that soak up performance) you'd probably be better off just spending the same time to do on a single machine.
Bonus, if your problem gets too big for a single machine (it happens), there might be trivial parallelism in the problem you can exploit and now going-wide means you'll probably outperform your original design anyways and the coordination code is likely to be much simpler and less performance degrading. Or you can go-high and toss more machine at it and get more gains with zero planning or effort outside of copying your code and the data to the new machine and plugging it in.
Oh yeah, many of us, especially experienced people or those with lots of school time, are taught to overgeneralize our approaches. It turns out many big compute problems are just big one-off problems and don't need a generalized approach. Survey your data, plan around it, and then write your solution as a specialized approach just for the problem you have. It'll likely run much faster this way.
Some anecdotes:
- I wrote an NLP tool that, on a single spare desktop with no exotic hardware, was 30x faster than a 6-high-end-system-distributed-compute-node that was doing a comparable task. That group eventually used my solution with a go-high approach and runs it on a big multi-core system with as fast of memory and SSD as they could procure and it's about 5 times faster than my original code. My code was in Perl, the distributed system it competed against was C++. The difference was the algorithm I was using, and not overgeneralizing the problem. Because my code could complete their task in 12 hours instead of 2 weeks, it meant they could iterate every day. A 14:1 iteration opportunity made a huge difference in their workflow and within weeks they were further ahead than they had been after 2 years of sustained work. Later they ported my code to C++ and realized even further gains. They've never had to even think about distributed systems. As hardware gets faster, they simply copy the code and data over and realize the gains and it performs faster than they can analyze the results.
Every vendor that's come in after that has been forced to demonstrate that their distributed solution is faster than the one they already have running in house. Nobody's been able to demonstrate a faster system to-date. It has saved them literally tens of millions of dollars in hardware, facility and staffing costs over the last half-decade.
- Another group had a large graph they needed to conduct a specific kind of analysis on. They had a massive distributed system that handled the graph, it was about 4 petabytes in size. The analysis they wanted to do was an O(N^2) analysis, each node needed to be compared potentially against each other node. So they naively set up some code to do the task and had all kinds of exotic data stores and specialized indexes they were using against the code. Huge amounts of data was flying around their network trying to run this task but it was slower than expected.
An analysis of the problem showed that if you segmented the data in some fairly simple ways, you could skip all the drama and do each slice of the task without much fuss on a single desktop. O(n^2) isn't terrible if your data is small. O(k+n^2) isn't much worse if you can find parallelism in your task and spread it out easily.
I had a 4 year old Dell consumer level desktop to use so I wrote the code and ran the task. Using not much more than Perl and SQLite I was able to compute a large-ish slice of a few GB in a couple hours. Some analysis of my code showed I could actually perform the analysis on insert in the DB and that the size was small enough to fit into memory so I set SQLite to :memory: and finished it in 30 minutes or so. That problem solved, the rest was pretty embarrassingly parallel and in short order we had a dozen of these spare desktops occupied running the same code on different data slices and finishing the task 2 orders of magnitude than what their previous approach had been. Some more coordinating code and the system was fully automated. A single budget machine was theoretically now capable of doing the entire task in 2 months of sustained compute time. A dozen budget machines finished it all in a week and a half. Their original estimate on their old distributed approach was 6-8 months with a warehouse full of machines, most of which would have been computing things that resulted in a bunch of nothing.
To my knowledge they still use a version of the original Perl code with SQlite running in memory without complaint. They could speed things up more with a better in-memory system and a quick code port, but why bother? It's completing the task faster than they can feed it data as the data set is only growing a few GB a day. Easily enough for a single machine to handle.
- Another group was struggling with handling a large semantic graph and performing a specific kind of query on the graph while walking it. It was ~100 million entities, but they needed interactive-speed query returns. They had built some kind of distributed Titan cluster (obviously a premature optimization).
Solution, convert the graph to an adjacency matrix and stuff it in a PostgreSQL table, build some indexes and rework the problem as a clever dynamically generated SQL query (again, Perl) and now they were realizing .01second returns, fast enough for interactivity. Bonus, the dataset at 100m rows was tiny, only about 5GB, with a maximum table-size of 32TB and diskspace cheap they were set for the conceivable future. Now administration was easy, performance could be trivially improved with an SSD and some RAM and they could trivially scale to a point where dealing with Titan was far into their future.
Plus, there's a chance for PostgreSQL to start supporting proper scalability soon putting that day even further off.
- Finally, a e-commerce company I worked with was building a dashboard reporting system that ran every night and took all of their sales data and generated various kinds of reports, by SKU, by certain number of days in the past, etc. It was taking 10 hours to run on a 4 machine cluster.
A dive in the code showed that they were storing the data in a deeply nested data structure for computation and building and destroying that structure as the computation progressed was taking all the time. Furthermore, some metrics on the reports showed that the most expensive to compute reports were simply not being used, or were being viewed only once a quarter or once a year around the fiscal year. And cheap to compute reports, where there were millions of reports being pre-computed, only had a small percentage actually being viewed.
The data structure was built on dictionaries pointing to other dictionaries and so-on. A quick swap to arrays pointing to arrays (and some dictionary<->index conversion functions so we didn't blow up the internal logic) transformed the entire thing. Instead of 10 hours, it ran in about 30 minutes, on a single machine. Where memory was running out and crashing the system, memory now never went above 20% utilization. It turns out allocating and deallocating RAM actually takes time and switching a smaller, simpler data structure makes things faster.
We changed some of the cheap to compute reports from being pre-computed to being compute-on-demand, which further removed stuff that needed to run at night. And then the infrequent reports were put on a quarterly and yearly schedule so they only ran right before they were needed instead of every night. This improved performance even further and as far as I know, 10 years later, even with huge increases in data volume, they never even had to touch the code or change the ancient hardware it was running on.
It seems ridiculous sometimes, seeing these problems in retrospect, that the idea was that to make these problems solvable racks in a data center, or entire data centeres were ever seriously considered seems insane. A single machine's worth of hardware we have today is almost embarrassingly powerful. Here's a machine that for $1k can break 11 TFLOPS [1]. That's insane.
It also turns out that most of our problems are not compute speed, throwing more CPUs at a problem don't really improve things, but disk and memory are a problem. Why anybody would think shuttling data over a network to other nodes, where we then exacerbate every I/O problem would improve things is beyond me. Getting data across a network and into a CPU that's sitting idle 99% of the time is not going to improve your performance.
Analyze your problem, walk through it, figure out where the bottlenecks are and fix those. It's likely you won't have to scale to many machines for most problems.
I'm almost thinking of coming up with a statement: Bane's rule, you don't understand a distributed computing problem until you can get it to fit on a single machine first.
The average of one murder per murderer I'd generally agree with, but one of the Eton murderers killed 9 before committing suicide. So you are right to have qualms. But it is an upper bound and so still useful.
That said, one correction. El Salvador had a murder rate in 2017 of 61.7/100,000. This is comparable to Eton.
Let's take your analysis a step farther. If we have 230 * 30 people, each of whom has 1.1 chances in 100,000 of committing murder, the number of expected murderers is 0.0759. And the distribution of number of murders is a Poisson distribution. That means that the probability of k murders is λ^k e^(-λ) / k!.
Add those up and the expected probability of 4 or more murderers is only 1.3014245782150269e-06.
Therefore, even with a small absolute number, we can be very sure that the true murder rate for graduates of Eton are significantly higher than the UK population.
However Eton does attract people internationally. And the international murder rate in 2017 was 6.1/100,000 per https://worldpopulationreview.com/country-rankings/murder-ra.... Using that as a murderer rate, the odds against Eton producing 4 murderers in 30 years improve...to about 1/1000. Which means that Eton's murders are still likely to not be chance.
Former PR & Ad agency owner here. Better to release on Thursday if you care about external impact. Better to release on Friday if you are concerned with internal impact.
Something many in business don't understand is Friday is the start of the weekend news cycle. If whatever bed news happens is released on Friday, and it is juicy, it will end up kicking around on weekend news shows, Sunday talk shows, Sunday print newspapers (which still matter in some large markets), and as a bonus, will cycle into the next week's news via follow ups. The reason to release on Friday is simple: you have the weekend to communicate with your key managers and do internal damage control (even if that is just keeping the team productive). For some CEOs, you turn off the phone, and go hit the Yacht for the weekend and hide. If you release bad news on Thursday, you'll potentially have less coverage, but have to be very adroit with your internal communication with employees. For CEOs that can't handle bad news, Thursday releases are really bad...
Learning how to speak chicken is not becoming a chicken. That's a frequent misconception that borders cultural appropriation. Chickens aren't just about a few bock-bock and cockle-doodle-doos. They have a rich set of physical expressions as well (flap flaps, head kicks, grating the soil, etc) and a vibrant culture that is transmitted between generations. Their philosophy on death and their capacity to always see life on the sunny-side is both unique and deep, the fruit of seeing their children being picked for food on a daily basis. This should be celebrated. The article completely missed the point on that aspect.
Pretending you became a chicken because you're now performing a few chicken-like sounds is plain wrong, an insult to the poultry community.
> How much do you think the instant gratification society that we're living in plays a role in this behavior?
Yea, my theory is that for some people (like myself), the instant-gratification society has been terrible for mental health. I have effectively retrained my brain to be stimulated by only exciting + novel things. Things that illicit huge dopamine releases due to their nature being inherently novel/new.
Take YouTube as an example. Pull down on the page, and it's like a slot machine. Page loads a whole new grid of fresh - potentially click worthy - videos!
The eyes/brain scan the page looking for a hit. BINGO you see a video that catches your interest. Click. Ahh, satisfying transition instantly to the video. Boring? Click back at the home page, and start the slot machine again. Rinse, repeat.
That cycle of chance (load the grid) --> seek (search the grid visually) --> instant reward (click the video) is what's so dangerous. Takes almost no mental, or physical effort, to carry out this cycle.
IMO, it's exploiting a very primitive system:
chance (randomness) --> stimulates effort --> get reward.
You will naturally tend toward systems that minimize effort, and maximize reward. Everything else will feel like a giant burden, or will lack a stimulating effect entirely.
You can see this same cycle in things like Facebook, Instagram, Google News, or any video game. Minimal effort required, to capture your attention (effectively blasting dopamine).
I hate all these click-bait articles. Losing weight is simple, it's just CICO. Calories In, Crust of the earth Out
Edit: I feel like a lot of people are arguing about something that should be considered totally uncontroversial, basic physics. Before you get all worked up over this, please look up Yuri Gagarin. His weight loss journey absolutely proved that it is possible to lose 100% of your body weight through simple orbital mechanics and hundreds of people have followed in his footsteps and experienced similar results. There is not room for debate, this is a proven weight loss method and what the I'm suggesting is essentially a very similar thing, but one that could help a lot more people at once.
My method sure hasn't worked very well... or has it?
I was looking for cheap warehouse space to start a business in, and did a nation-wide search for the largest, cheapest building in the entire continental US.
I found one that seemed too good to be true. a 220,000 sqft metal warehouse and office complex on 17 acres. I thought the price was a typo at $375k.The agent assured me that the price was correct, and I flew out to see the place.
It was in a little town called Pine Bluff, Arkansas.
I offered about 3/4 what they were asking, and they accepted the offer.
Fast forward 2 1/2 years, and I've had nothing but problems. Break in after break in. Can't work through the red tape with the city so my warehouse sits empty. It feels like they are actively working against myself and other entrepreneurs I talk to. At least 2 others who bought buildings and tried to open businesses left after getting nowhere.
Maybe I'm daft, but I ended up buying about 75 more properties here... all surprisingly cheap.
The town is killing me though. I haven't seen my kids very much lately - I don't think it's safe enough for them. I'm probably going to be moving back to Utah in the next couple of months because it's just too much out here.
If you're tempted to use audit tables, you might also consider making the jump to temporal tables. I rolled my own light-weight version using inheritance-based partitioning in Postgres. The basic idea is:
- Create a parent table like items with a valid time range column as a tstzrange type. This table won't store data.
- Create two child tables using Postgres inheritance, item_past and item_current that will store data.
- Use check constraints to enforce that all active rows are in the current table (by checking that the upper bound of the tstzrange is infinite). Postgres can use check constraints as part of query planning to prune to either the past or current table.
- Use triggers to copy from the current table into the past table on change and set the time range appropriately.
The benefits of this kind of uni-temporal table over audit tables are:
- The schema for the current and past is the same and will remain the same since DDL updates on the parent table propagate to children. I view this as the most substantial benefit since it avoids information loss with hstore or jsonb.
- You can query across all versions of data by querying the parent item table instead of item_current or item_past.
The downsides of temporal tables:
- Foreign keys are much harder on the past table since a row might overlap with multiple rows on the foreign table with different times. I limit my use of foreign keys to only the current table.
> This is very reminiscent of Google/YouTube circa 2006.
In that it was a startup acquired by a big company? Yep.
> When Google bought YT it was a small team of people and a pretty nascent product that people really loved, and the usage numbers were out of control.
Like most startup acquisitions, the team size is relatively small and there is significant traction in the market with headroom to mature their footprint.
> They left the product mostly untouched and let it grow on its own. Though there was major criticism at the time, it is one of the best tech acquisitions of the past decade.
This is not going to be one of the best tech acquisitions of the next decade. YouTube helped to propel Google into content. It also helped to commoditise web video in a massive way: reminiscent of the way which Google commoditised search (YouTube is probably just short of being a byword for online video at this point).
Instagram is a photo service in a sea of other photo services. Photography has been around on the web in meaningful ways for a long time. Flickr lost out to Facebook in the community stakes, and Instagram is doing great in whatever-the-fuck market it's in (the share-to-my-twitter-followers market?), but this is not Google acquiring YouTube.
Yes. The CPU and GPU demand has nothing to do with it. The reason is the car industry.
For some reason in early 2020 all the car industry execs were convinced that people would buy dramatically fewer cars in 2020, due to pandemic crashing demand. Because they have a religious aversion to holding any stock they decided to shift the risk over to their suppliers, fucking said suppliers over, as the car industry normally does when they expect demand shifts. The thing that made this particular time special as opposed to business as usual is that the car execs all got it wrong, because people bought way more cars due to pandemic rather than less, due to moving out of cities and avoiding public transit. So they fucked over their suppliers a second time by demanding all those orders back.
Now, suppose you're a supplier of some sort of motor driver or power conversion chip (PMIC) in early 2020. You run 200 wafers per month through a fab running some early 2000s process. Half your yearly revenue is a customized part for a particular auto vendor. That vendor calls you up and tells you that they will not be paying you for any parts this year, and you can figure out what to do with them. You can't afford to run your production at half the revenue, so you're screwed. You call up your fab and ask if you can get out of that contract and pay a penalty for doing so, and you reduce your fab order to 100 wafers per month, so you can at least serve your other customers. The fab is annoyed but they put out an announcement that a slot is free, and another vendor making a PMIC for computer motherboards buys it, because they can use the extra capacity and expect increased demand for computers. So far so normal. One vendor screwed, but they'll manage, one fab slightly annoyed that they had to reduce throughput a tiny bit while they find a new buyer.
Then a few months later the car manufacturer calls you again and asks for their orders back, and more on top. You tell them to fuck off, because you can no longer manufacture it this year. They tell you they will pay literally anything because their production lines can't run without it because (for religious reasons) they have zero inventory buffers. So what do you do? You call up your fab and they say they can't help you, that slot is already gone. So you ask them to change which mask they use for the wafers you already have reserved, and instead of making your usual non-automotive products, you only make the customized chip for the automotive market. And then, because they screwed you over so badly, and you already lost lots of money and had to lay off staff due to the carmaker, you charge them 6x to 8x the price. All your other customers are now screwed, but you still come out barely ahead. Now, of course the customer not only asked for their old orders back, but more. So you call up all the other customers of the fab you use and ask them if they're willing to trade their fab slots for money. Some do, causing a shortage of whatever they make as well. Repeat this same story for literally every chipmaker that makes anything used by a car. This was the situation in January 2021. Then, several major fabs were destroyed (several in Texas, when the big freeze killed the air pumps keeping the cleanrooms sterile, and the water pipes in the walls of the buildings burst and contaminated other facilities, and one in Japan due to a fire) making the already bad problem worse. So there are several mechanisms that make part availability poor here:
1. The part you want is used in cars. Car manufacturers have locked in the following year or so of production, and "any amount extra you can make in that time" for a multiple of the normal price. Either you can't get the parts at all or you'll be paying a massive premium.
2. The part you want is not used in cars, but is made by someone who makes other parts on the same process that are used in cars. Your part has been deprioritized and will not be manufactured for months. Meanwhile stock runs out and those who hold any stock massively raise prices.
3. The part you want is not used in cars, and the manufacturer doesn't supply the car industry, but uses a process used by someone who does. Car IC suppliers have bought out their fab slots, so the part will not be manufactured for months.
4. The part you want is not used in cars, and doesn't share a process with parts that are. However, it's on the BOM of a popular product that uses such parts, and the manufacturer has seen what the market looks like and is stocking up for months ahead. Distributor inventory is therefore zero and new stock gets snapped up as soon as it shows up because a single missing part means you can't produce your product.
So here we are. Shameless plug - email me if you are screwed by this and need help getting your product re-engineered to the new reality. There's a handful of manufacturers, usually obscure companies in mainland China that only really sell to the internal market, that are much less affected. Some have drop-in replacement parts for things that are out of stock, others have functionally similar parts that can be used with minor design adaptation. I've been doing that kind of redesign work for customers this whole year. Don't email me if you work in/for the car industry. You guys poisoned the well for all of us so deal with it yourselves.
Your comment reminded me about a passage from Red Dwarf:
In fact, it was now possible for Rimmer to revise solidly for three months and not learn anything at all.
The first week of study, he would always devote to the construction of a revision timetable. At school Rimmer was always at his happiest colouring in geography maps: under his loving hand, the ice-fields of Europa would be shaded a delicate blue, the subterranean silica deposits of Ganymede would be rendered, centimetre by painstaking centimetre, a bright and powerful yellow, and the regions of frozen methane on Pluto slowly became a luscious, inviting green. Up until the age of thirteen, he was constantly head of the class in geography. After this point, it became necessary to know and understand the subject, and Rimmer's marks plunged to the murky depths of 'F' for fail.
He brought his love of cartography to the making of revision timetables. Weeks of patient effort would be spent planning, designing and creating a revision timetable which, when finished, were minor works of art.
Every hour of every day was subdivided into different study periods, each labelled in his lovely, tiny copperplate hand; then painted over in watercolours, a different colour for each subject, the colours gradually becoming bolder and more urgent shades as the exam time approached. The effect was as if a myriad tiny rainbows had splintered and sprinkled across the poster-sized sheet of creamwove card.
The only problem was this: because the timetables often took seven or eight weeks, and sometimes more, to complete, by the time Rimmer had finished them the exam was almost on him. He'd then have to cram three months of astronavigation revision into a single week. Gripped by an almost deranging panic, he'd then decide to sacrifice the first two days of that final week to the making of another timetable. This time for someone who had to pack three months of revision into five days.
Because five days now had to accommodate three months' work, the first thing that had to go was sleep. To prepare for an unrelenting twenty-four hours a day sleep-free schedule, Rimmer would spend the whole of the first remaining day in bed – to be extra, ultra fresh, so he would be able to squeeze three whole months of revision into four short days.
Within an hour of getting up the next morning, he would feel inexplicably exhausted, and start early on his supply of Go-Double-Plus caffeine tablets. By lunchtime he'd overdose, and have to make the journey down to the ship's medical unit for a sedative to help him calm down. The sedative usually sent him off to sleep, and he'd wake up the following morning with only three days left, and an anxiety that was so crippling he could scarcely move. A month of revision to be crammed into each day.
At this point he would start smoking. A lifelong non-smoker, he'd become a forty-a-day man. He'd spend the whole day pacing up and down his room, smoking three or four cigarettes at a time, stopping occasionally to stare at the titles in his bookcase, not knowing which one to read first, and popping twice the recommended dosage of dog-worming tablets, which he erroneously believed to contain amphetamine.
Realizing he was getting nowhere, he'd try to get rid of his soul-bending tension by treating himself to an evening in one of Red Dwarf's quieter bars. There he would sit, in the plastic oak-beamed 'Happy Astro' pub, nursing a small beer, grimly trying to be light-hearted and totally relaxed. Two small beers and three hours of stomach-knotting relaxation later, he would go back to his bunk and spend half the night awake, praying to a God he didn't believe in for a miracle that couldn't happen.
Two days to go, and ravaged by the combination of anxiety, nicotine, caffeine tablets, alcohol he wasn't used to, dog-worming pills, and overall exhaustion, he would sleep in till mid-afternoon.
After a long scream, he would rationalize that the day was a total write-off, and the rest of the afternoon would be spent shopping for the three best alarm clocks money could buy. This would often take five or six hours, and he would arrive back at his sleeping quarters exhausted, but knowing he was fully prepared for the final day's revision before his exam.
Waking at four-thirty in the morning, after exercising, showering and breakfasting, he would sit down to prepare a final, final revision table, which would condense three months of revision into twelve short hours. This done, he would give up and go back to bed. Maybe he didn't know a single thing about astronavigation, but at least he'd be fresh for the exam the next day.
Suppose the real number is 3%. 74% sounds pretty good as a way to soften the blow a bit. It would be best to let the air out over a matter of weeks than minutes.
Now let's take this to the extreme: What if you mined cryptocurrency by burning medicine? The more you burn, the richer you are. Clearly that's evil, as it rewards you for destroying the world.
In a perfect world you would mine cryptocurrency by making progress in peace on earth, good will towards man.
But if you can answer this: What useful resource, that's expensive to produce and has an environmental impact, do you want to incentivize destroying in order to further your greed?
Can you think of anything? Is it not by its very nature evil?
But consider even what happened under Imperial rule in India: Locals were rewarded for every dead snake they proved to the British. What happened? Obviously they started breeding snakes!
So now you have MORE snakes! You actually made the problem worse!
The problem with analogizing Bitcoin mining to gold mining is that Bitcoin mining secures value in the past and gold mining secures value in the future. If you stopped mining Bitcoin, all previously mined Bitcoin would lose value due to the decreased security of the block chain.
It's like if gold began to sink into the ground the moment you stopped mining it, and you had to keep mining and smelting it at a ever-more furious rate to keep it above ground.
Is there a name for the opposite effect? If not, I propose the "Sinclair Effect" after "It is difficult to get a man to understand something, when his salary depends on his not understanding it."
You open the newspaper to an article on some subject that you know well, because your salary depends on it. In Tourre's case, finance. In mine, tech. You read the article and see that the journalist isn't a subject matter expert and has made an error about some fact. Often, the error is so small that it doesn't affect the story at all. I call these the "well, actually, some streets are sheltered from the rain" stories. In any case, you dismiss the entire story and stop reading it, and then turn the page to national or international affairs, and read as if your technical expertise somehow gave you insight into a place you've never been and a culture you've never interacted with. You turn the page, and forget the gaps in your own knowledge.
Lets say you were just hired as the President of a furniture company. The owner says he knows it's good furniture but even despite huge investments they can't seem to sell any furniture. Your job is to turn things around.
You start on the factory floor. The furniture is made by a combination of machines and human workers. Some people are employed to set up and configure the machines to make furniture parts. Around 150 people work on actually making furniture, either assembling it, doing quality tests, or setting up and operating the automated machinery. Things aren't perfect, but you aren't going to make any changes on your first day so you make some notes and move on. The furniture hasn't changed much over the years, it is still basically the same as it was when the furniture store opened. The furniture gets 'improved' from time to time, you see a step stool with an alarm clock, a small safe, and a webcam built into it, but when you ask the foreman he tells you nobody has ever turned on the alarm clock or used the safe or connected the webcam on any of the step stools. People seem to mainly use the stools so they can reach things that are up high. There is a problem where sometimes people slip when the stools are wet, so they worked out how to add a nonslip pad, but the product managers have decided that the next feature will be to add scents to the stools, so you can buy a stool that smells like cinnamon or one that smells like apples. They have a big advertising campaign already paid for and they already sent out the press release announcing "ScentedStools", so the machines need to be set up to start stamping out stools that smell like "Fresh Linen" by the end of the week. There are daily status meetings to update them on the progress. If the "Fresh Linen" stools aren't being produced by Thursday they are going to start having two status meetings per day.
You hear it's someone named Jim's last day, so you set up an exit interview. Jim tells you that the bosses and people upstairs don't really know what is going on in the factory. Most days he just sits and reads the news, his "nontechnical" manager doesn't know anything about furniture or how Jim does his job so there's no way for the manager to know what is going on other than to ask Jim. Supervision primarily consists of making sure Jim is sitting at his desk and looking at his monitor.
Since US labor laws don't allow Jim's manager to set specific hours for him to be at work, his manager has started scheduling 9AM meetings every day to force people to turn up. Every week or so Jim has to update some Product Managers upstairs about what is going on, and he just says they are making steady progress and comes up with some specific problem to explain why they aren't done, pretty much anything with jargon will work since nobody upstairs "could tell white oak from red oak". It takes about 5 minutes to give his status update but he's expected to stay for the entire 1 hour meeting, so he brings his laptop so he can read furniturenews.com. He says he is quitting to take a much lower paying job because he is bored and doesn't respect his manager.
Next you go upstairs to the office space and find 300 people having meetings with each other about annual plans and prioritization, writing mission statements and meeting to discuss mission statements. The 300 people upstairs are constantly in motion and complaining about how over worked they are. They each have 5, 6 or even 7 (sometimes more!) 1-hour meetings every day, but you only see them meet with each other, nobody has any meetings with anyone from outside the company, nobody has meetings with possible customers, and only very rarely do you see anyone from the factory floor in these meetings, and then it is almost always just to give a status update. None of these folks really understand furniture very well, they can't really tell good furniture from bad furniture, they literally don't know the difference between solid oak and cardboard, they don't know how long it takes or how much money it costs to build a chair. After a few days of meetings you haven't met anyone who cares about furniture at all, they all seem to want to work at the furniture factory because it pays well, or they like the prestige of being 'in furniture'. Mostly they talk about how overworked they are and make the case for hiring a few more people. If they could hire another person for their team they wouldn't be so far behind. You aren't sure what they are getting behind in, are they talking about meetings they can't attend because it conflicts with another meeting that is more important somehow? Do they need more time to work on power point slides for the next days meetings? Some of the office folks have degrees in furniture science, but none of them have ever successfully built or designed any furniture outside of little school projects.
Then you go out behind the factory and see a massive mountain of furniture stacked up to the sky. The factory workers have been building furniture every day for years. People all agree that it is good furniture, maybe the best there is. Nobody ever buys any of it. It's not sold in any stores. No hotels buy it. No businesses buy it. Lots of people are lined up as far as you can see to pick furniture out of the pile for free.
Unimation had the first 2-arm robot, a research project for Ford. I was in charge of programming the Nova (Data-General) computer (by toggle swithes first, and then by paper tape).
We were assembling a transmission control unit. It changed gears based on rotation speed by sliding a calibrated spool valve against a spring. The spool valve was a TIGHT fIt. It controlled oil flow to shift the transmission. You couldn't insert it by hand without it jamming. The robot was so precise it could insert it easily.
The robot was made of aluminum and had to be run until it reached operating temperature because otherwise the joint lengths were not right. Everything had to be perfect.
Assembling involved picking up the control unit, placing a small cover, driving two screws, turning the controller over, inserting a spring, inserting the spool valve, placing a second cover, inserting two screws, and put the finished assembly away. Repeat.
We made hundreds of these. The robot was FAST. REALLY FAST. In a fit of misplaced confidence we invited Ford senior executives to a demo (I HATE THAT WORD). They had invested a LOT of money so I guess they should see what they bought.
On the day of the demo, during warmup, the robot managed to mangle its fingers on a fixture. Our machinist made a new set from raw materials while we smiled and chatted with the Ford executives and warmed up the machine.
Demo time. We lined up near the robot, with the Unimation chief engineer and the project engineer, and the 3 Ford senior execs in front, Ming and I behind and out of the way.
Push the button.... The robot picks up the control unit, places the small cover, throws the screws at the Ford people, turns the controller over, throws the spring at the Ford people, picks up the spool valve (a solid steel piece) and... We hit emergency stop (the big red button). I nearly died trying to keep from laughing watching Ford execs dodging flying parts.
We recalibrated the robot and re-ran the demo, which worked fine.
A fun and fairly simple project, with a surprisingly high ratio of usefullness to effort, is to write an interpreter for a concatenative language. Concatenative languages, like FORTH, can do a lot with very limited resources, making them good candidates for embedded systems.
If you want to play around with making your own concatenative language, it is actually surprisingly simple. Here is an overview of a step-by-step approach that can take you from a simple calculator to a full language with some optimization that would actually be quite reasonable to use in an embedded system.
So let's start with the calculator. We are going to have a data stack, and all operations will operate on the stack. We make a "dictionary" whose entries are "words" (basically names of functions). For each word in the dictionary, the dictionary contains a pointer to the function implementing that word.
We'll need six functions for the calculator: add, sub, mul, div, clr, and print. The words for these will be "+", "-", "x", "/", "clr", and "print". So our dictionary looks like this in C:
We need a main loop, which will be something like this (pseudocode):
while true
token = NextToken()
if token is in dictionary
call function from that dict entry
else if token is a number
push that number onto the data stack
Write NextToken, making it read from your terminal and parse into whitespace separated strings, implement add, sub, mul, div, clr, and print, with print printing the top item on the data stack on your terminal, and you've got yourself an RPN calculator. Type "2 3 + 4 5 + x print" and you'll get 45.
OK, that's fine, but we want something we can program. To get to that, we first extend the dictionary a bit. We add a flag to each entry allowing us to mark the entry as either a C code entry or an interpreted entry, and we add a pointer to an array of integers, and we add a count telling the length of that array of integers. When an entry is marked as C code, it means that the function implementing it is written in C, and the "func" field in the dictionary points to the implementing function. When an entry is marked as interpreted, it means that the pointer to an array of integers points to a list of dictionary offsets, and the function is implemented by invoking the functions of the referenced dictionary entries, in order.
A dictionary entry now looks something like this:
struct DictEntry {
char * word;
bool c_flag;
void (*func)(void);
int * def;
int deflen;
}
If it is a single-issue negotiation (i.e. know your own bottom line and there is no BATNA https://en.wikipedia.org/wiki/Best_alternative_to_a_negotiat... other than this one choice), therefore you're not afraid to lose (i.e. there are lots of potential suitors), then bargain like a maniac. The only thing they have to hold you by is a time-limited offer, but by doing this they've revealed part of their own non-bottom line (i.e. fear of losing is part of their BATNA), so explore this.
Have recommended before and recommend again the book 'Getting to Yes', and Harvard's very open research and sharing in this area: http://www.pon.harvard.edu/
Getting too serious? Not at all. Harvard's role-plays that can be purchased very cheaply ($1-3 per copy) are great teamwork activities.
They tend to focus on numerical amounts denominated in dollars, but these financial numbers can be easily substituted for time, number of people, anything that's a number. Practicing new concepts with things you're immediately familiar with tends to lead to remembered solutions. Instead save the comparison and application to the postmortem.
What the OP takes to the extreme is a property of weak typing, not dynamic typing.
These properties are separate issues. A language is either statically typed or dynamically typed, and it is also either strongly typed or weakly typed.
In a statically typed language, types are attached to variables. In a dynamically typed language, types are attached to values. Do note that many languages don't fit 100% into either category. (Note that the blog post you linked to calls types attached to values classes, but that distinction often isn't made so clearly.)
A weakly typed language performs implicit type conversions as it deems necessary, while a strongly typed language does no implicit type conversions. Most languages don't fit 100% into either category. Usually languages that are considered to be strongly typed still allow you to add integers to floating point numbers, for example.
It is possible for a dynamically typed language to be so strongly typed that it won't do any implicit type conversion, ever. Such a language would not allow you to, say, add an integer and a floating point number without explicitly converting one to the other.
It is also possible for a statically typed language to be so weakly typed that it implicitly converts types everywhere. Such a language might do the exact same things the OP uses, like converting a function to a string of its source code when adding it to a string.
So I have a theory that may explains things here. Here it is.
There is 3 levels of programmers.
Level 1 is the beginner. A level 1 programmer is barely able to make a complete app or lib. His work simply rarely works. The code has no sense, is full of bugs, it's a complete mess, and the user has a very poor experience (if an experience at all).
Then comes the level 2 programmer. The novice. The novice learnt about OOP, design patterns, DRY, single responsability principle, importance of testing, etc. The only problem with the level 2 programmer is that he overdoes everything. He is victim of overengineering. He is able to make a complete app with a decent UX and the quality of the code is dramatically better, but his productivity is low. Really, really low. Everything takes month. Every single change is a project. 10 actual lines of codes need to be changed? That's 10 classes, 250 lines of unit tests, it requires 3 classes to be refactored, and on and on.
Finally, there is the level 3 programmer. The level 3 programmer is a programmer that does level 2 programming in a smarter way. His first principle is less code. That doesn't mean the code must be complete shit, but the level 3 engineer understand the real enemy and that all problems grow with it exponentially: code. The level 3 programmer knows, understand and apply the principles of the level 2 programmer, but he just does the right amount of them. Not too much.
If he has to, the level 3 programmer gonna err on the side of the level 1 code and will avoid as much as he can level 2 code. That's because level 1 code cost less to fix than level 2 code.
Now here comes my point: a level 2 programmer read and write articles that is about how to not be a level 1 programmer.
I have been advocating for the PrefixVarint encoding you mention for a while.
One thing I'd mention though: as you've specified it here, it puts the continuation bits as the high bits of the first byte. I think it may be better to put them in the lower bits of that byte instead. It would allow for a simple loop-based implementation of the encoder/decoder (LEB128 also allows this). With continuation bits in the high bits of the first byte, you pretty much have to unroll everything. You have to give each length its own individual code-path, with hard-coded constants for the shifts and continuation bits.
The downside is one extra shift of latency in the one-byte case, imposed on all encoders/decoders.
Unrolling is probably a good idea for optimization anyway, but it seems better to standardize on something that at least allows a simple implementation.
Here is some sample code for a loop-based implementation that uses low bits for continuation bits:
// Little-endian only. Untested.
char *encode(char *p, uint64 val) {
int len = 1;
uint64 encoded = val << 1;
uint64 max = 1 << 7;
while (val > max) {
if (max == 1ULL << 63) {
// Special case so 64 bits fits in 9 bytes.
*p++ = 0xff;
memcpy(p, &val, 8);
return p + 8;
}
encoded = (encoded << 1) | 1;
max <<= 7;
len++;
}
memcpy(p, &encoded, len);
return p + len;
}
const char *decode(const char *p, uint64* val) {
if (*p == 0xff) {
// 9-byte special case
memcpy(val, p + 1, 8);
return p + 9;
}
// Can optimize with something like
// int len = __builtin_ctz(!*p);
unsigned char b = *p;
int len = 1;
while (b & 1) {
len++;
b >>= 1;
}
*val = 0;
memcpy(val, p, len);
*val >>= len;
return p + len;
}
Here's an illustration; this may be somewhat dated (not sure how this works with e.g. shared L3 caches) but it should give you a flavor of why shared memory sucks for messaging.
Consider the simplest example of a busy-loop, which is likely close to the fastest you can get. Core1 sends a message to Core2 by writing to an address that Core2 is repeatedly reading from.
Core1 first has to gain exclusive access to the address, by broadcasting an invalidate on the bus. Core2 thus discards its cache line. Next Core1 writes to the address by modifying the data in its L1 cache (or store buffer or worse). Core2 then reads from the address. Since Core1 holds the modified cache line, it is required to snoop the read. Core1 tells Core2 to wait and then retry. This happens repeatedly until Core1's write lands in main memory. Now Core2 can read the memory.
So:
1. Messaging requires a round trip to memory, a bunch of cache line invalidation, and other nonsense. Messaging involves multiple transitions to and from shared state.
2. Your CPUs already have a very fast message bus to implement MESI, but it is unavailable to you as a client programmer.
It should be obvious from this that we could build a much faster CPU messaging architecture if we cared to.
IMO an appeal to authority is a perfectly reasonable response to an assumption of incompetence.
It's tiresome to always see comments assuming that such-and-such science is flawed because a mainstream news article doesn't spell out every single detail.
If it were phrased more neutrally, like "I'm amazed that this stuff can be contained without any contamination for hundreds of millions of years, how do they know that it works that way?" I wouldn't have a problem with it. Start assuming that you've missed something, not the scientist who does this for a living. Maybe the scientist has missed something, they're certainly not perfect, but to me it's ridiculous and counterproductive to be "worried" about the results of years of professional work based on one popular article and five minutes of thought (if that).
Wow. That third message, with suggestions for avoiding the bug, reads like a twisted joke. Highlights:
"avoid struct types which contain both integer and pointer fields"
"avoid data structures which form densely interconnected graphs at run-time"
"avoid integer values which may alias at run-time to an address; make sure
most integer values are fairly low (such as: below 10000)"
I understand that this isn't a completely brain-dead garbage collector, but warnings like that really scream "I'm just a toy language". It doesn't seem wise to call such a fragile programming tool production-ready or 1.0; the 32-bit implementation should be tagged as experimental, if only to lessen the damage to Go's reputation.
https://www.latacora.com/blog/2020/03/12/the-soc-starting/
https://fly.io/blog/soc2-the-screenshots-will-continue-until...