This is a pretty hilarious and long-winded way to say "we have no idea how to lock someone out of a web service:"
> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.
> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.
Right?! Did nobody there think to actually disable the accounts? These are the people who are harping about "security" being the reason for the ham-fisted takeover of the source repos, but they didn't secure the production infrastructure?
No matter how you slice it this is miserable root password security. Why do maintainers need root access? No one in my org has root access but me and all those creds are tied to hardware MFA locked in my MDF.
Those tools are certainly useful! In the recent production incident I mentioned, we we able to quickly use SLOWLOG to determine which calls were causing the latency spikes. Thanks deeply for providing all of these useful tools and docs.
> there is a percentage of users that will not read the doc and just deploy it
I wonder what this percentage is? I'd wager that it is higher than you might have anticipated, especially given the other comments on this thread.
As an engineer on a team ultimately responsible for the availability of a production service, it's my responsibility to ensure that the percentage of engineers that know the latency side effects of any Redis calls they make is near 100%. In the presence of such variable latency, any means of making that variability more obvious to all users of Redis would be a positive step towards happy users and operators.
I understand your point of view. Unfortunately I think Redis in this regard is a tool where to help is hard: I try to provide documentation but is the kind of tool that looks superficially so easy to use, yet you need some understanding in order to really use it effectively and deploy it safely. It's part of the fact that uses 1) uncommon tradeoffs and 2) is a set of commands without a predefined use-case, so there is tons to invent in the good and bad side :-)
This line jumped out to me as well. After a recent production incident determined to have been heavily influenced by several thousand O(N) commands being called against a list several orders of magnitude larger than normal, I'm thinking about this a lot.
I'm confident many other users of Redis are using O(N) operations in production against small datasets without knowledge of how much latency will be introduced by those operations when that dataset grows. This is exactly the kind of situation that makes me immediately skeptical when I find Redis in a emergent system design.
I'm considering what initially felt like a draconian means of remediation: using rename-command to rename all O(N) commands to BIG_O_N_$COMMAND to ensure everyone using them knows the possible impact and to allow for easy detection during code review and/or Redis latency spikes.
The more I think about it this approach, the more I feel that this should be the default mode of operation for Redis in production. SREs around the world would collectively save decades of time if the every engineer writing Redis queries to knew this fact by heart:
> Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes.
I threw out that prototype soon after the talk. At the time, there weren't a lot of other engineers at the company doing Erlang, so maintenance was considered to be a long-term problem. I'm glad we made that call.
There are so many presentations of the form "We're using Erlang/Scala/whatever, it's so awesome!", but so few followups when they give up on the idea for production..
It's hard to sustain some alternate technology in the face of common knowledge. Rarely does technical advantage outweigh hard-won operational experience.
As much of a fan of Erlang and Riak as I am, you did make the right call. If you only have one or two people on the team who want to know that technology, then it isn't a smart move to base a core piece of technology on it (Erlang) just because it might be the best answer. Sometimes an okay answer that everyone is familiar with is much better.
Hi cdibona! I'm primary on-call at GitHub today, and I wanted to say thanks to everyone that worked on https://code.google.com/export-to-github/ and worked with us to load test it before this announcement. It's really cool to see so much work put into making it easy for folks to easily move their data.
Government says cyber to differentiate from traditional "security", which means "establishing or maintaining control by putting people in cages or killing them". When talking with someone who can launch missiles, order soldiers to kill, or dispatch law enforcement officers to assault a crowd of students holding a protest, then saying cyber isn't ridiculous.
Computer security people just say "security" without any qualifiers, because none are required. When a salesman pitching an anti spam product says cyber, he's failing the shibboleth
Who cares? Honestly, who cares what word they use. We all know what they mean.
Comments like this one just seem to me like a form of avoidance: let's mock their vocabulary, because we don't want to think about the substance of what they're doing.
It's a term that you mostly find in government circles, media reporting and advertising. Outside of these, it is a warning sign that a person comes from that background and has not much down-to-earth experience with security topics and/or is trying to sell you something. (At least that is my impression from reading various discussions about the word)
A lot of my fellow security analysts have an irrational dislike of the term cyber. Personally, I'm indifferent towards the word. When someone uses it, I know exactly what they are talking about. Isn't that the important thing?
> 1. While Ruby Central correctly removed access to shared credentials through its enterprise password manager prior to the incident, our staff did not consider the possibility that this credential may have been copied or exfiltrated to other password managers outside of Ruby Central’s visibility or control.
> 2. Ruby Central failed to rotate the AWS root account credentials (password and MFA) after the departure of personnel with access to the shared vault.