I got contacted by our rep a couple weeks ago, who informed me of this news. I thought it was a disaster and it really pissed me off. The rep couldn't even explain the reasoning well. It basically summed up to "because we can" and "where are you going to go?". He was shocked to find out that I didn't like it.
We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:
- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.
- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]
- Still no SDK of any kind.
- "Actions Data Stream" - or you can just fix your logging.
There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.
"Thank you for your interest in this GitHub action, however, right now we are not taking contributions.
We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in."
Nearly 20 years ago, some VP at a security products company now owned by Broadcom threatened us during contract renewal with, "The price is what it is. Your contract is up in two weeks. What are you going to do? Move to a competing product?"
This kinda change also has some different gears turning in my head. At $0.002 / build-minute, some of our large software integration tests would cost us around 15 - 20 cents. Some of our ansible integration tests would be 5 - 10 cents - and we run like 50 - 100 of those per day. Some deployments might cost us a cent or two.
Apples to oranges, naturally, but like this, our infra-jenkins master would pay for itself in hosting in a week of ansible integration testing compared to what GHA would cost. Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
> Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.
Depending on your workplace, there's a whole extra layer of bureaucracy and compliance involved if you self-host things. I aggressively avoid managing any VMs for that reason alone.
Luckily, at work we are this layer of bureaucracy and compliance. I'm very much pushing the agenda and idea that managing a stateful, mutable linux VM is a complex skill on it's own and incurs toil that's both recurring and hard to automate. The best case to handle that is to place your use case into our config management and let us manage it.
Most modern development workflows should just pickup a host with some container engine and do their work in stateless containers with some external state mapped in, like package caches. It's much easier for both sides in a majority of cases.
> And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
This is kinda where I am. No one really feels like they are selling a premium "just works" product. Its all jank. So why it the jank I chose at the price I chose?
At the moment I'm self hosting gitlab runners. Its jank. But it's free.
A while ago I set out to find a replacement for Jenkins. On prem with a comparable feature set. What I found out is that Jenkins is the worst apart from all the others.
> And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.
Self-hosting Jenkins on an EC2 instance is probably going to result in a _better_ experience at this point. Github Cache is barely better than just downloading assets directly, and with Jenkins you can trivially use a local disk for caching.
Or if you're feeling fancy and want more isolation, host a local RustFS installation and use S3 caching in your favorite tools.
Self-hosting on a host whose data actually persists is an even better experience, as it removes a lot of the tedium and workarounds such as extracting/down-/up-loading caches and so on. Get another host for redundancy and call it a day.
Hardware is getting cheaper and cheaper, but the fear-mongering around running a Linux machine has successfully prevented most businesses from reaping those cost reductions.
I repurposed old M1/M4 Mac Mini's at my workplace into GitHub action runners. Works like a charm, and made our workflows simpler and faster. Persisting the working directory between runs was a big performance boost.
Complete persistence has its downsides, as you can start getting "path dependency". E.g. a build succeeds only because some images were pre-cached by a previous build.
But having an _option_ to not download everything every time is great. You can add a periodic cache flushing, after all.
Last place I worked had long running end to end tests that would take 30 minutes on GHA (compared to maybe 5 locally) on every PR. This is going to make that a very expensive endeavour
We host a fair bit of Terraform code in a repos on GitHub, including the project that bootstraps and manages our GH org’s config: permissions, repos, etc.
Hilariously, the official Terraform provider for GitHub is full of N+1 API call patterns — aka exponential scaling hotspots — so even generating a plan requires making a separate (remote, rate-limited) API call to check things like the branch protection status of every “main” branch, every action and PR policy, etc. As of today it takes roughly 30 minutes to do a full plan, which has to run as part of CI to make sure the pushed TF code is valid.
With this change, we’ll be paying once to host our projects and again for the privilege of running our own code on our own machines when we push changes…and the bill will continue to grow exponentially b/c the speed of their API serves to set an artificial lower bound on the runtime of our basic tests.
(To be fair, “slow” and “Terraform” often show up and leave parties at suspiciously similar times, and GitHub is far from the only SaaS vendor whose revenue goes up when their systems get slower.)
It seems clear to me this is in response to all the third party GHA runners who were undercutting GitHub by just reselling cloud instances for cheaper.
They’ve lowered their runner costs to compete, and introduce minimum charge to discourage abd make sure they still get paid.
And if you want any concurrency at all, you need 1 runner registration per concurrent job. And each runner needs its own user. And each runner requires a full and separate copy of the runner software, which is large (hundreds of megs) and self-updates.
Ah right, I've forgotten because I'm using a multi-user strategy and a patched version of the runner at this point anyway. The config directory for each runner is normally based on its install path (insane), something like that?
I am developing a self-hosted solution for this [1]. It’s true that it’s somewhat of a pain but JIT runners allow a lot of flexibility that we don’t find elsewhere.
It will really depend on your workflows, what services you use, what risks you're trying to manage and what trade-offs you're willing to accept on usability vs security.
For some scenarios, resource-based policies can form the foundation of your auth flow. If you look at the flows described in the docs [1], resource evalutionn is simpler. You still need to solve the problem of effectively managing all of those resource policies and limitations on where they can be applied [2]. That might be an easier problem to solve then dealing with trying to express everything as an identity policy. You're then less concerned with wider permissions at the IAM level and move the responsibility to the owner of the resource.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York
Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standardize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
Foursquare | Devops/SRE/Systems Engineering (multiple roles) | Onsite | Full-time | Los Angeles, Chicago, Seattle, New York
Foursquare is the leading independent location technology platform, powering business solutions and consumer products through a deep understanding of location. Over 1,000 clients—including more than 50% of the Fortune 100—choose Foursquare.
We are looking for multiple roles, including Lead/Sr/Manager. Our stack is pretty wide, and includes AWS, K8s, ECS, Colo, Hadoop Ecosystem, observability platforms and quite a few other things. We've recently had a merger and an acquisition and a lot of work immediately in front of us is to unify and standarize across multiple technology stacks.
We have a large AWS presence and are responsible for both the data pipeline platforms as well as high traffic consumer-facing applications.
We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:
- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.
- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]
- Still no SDK of any kind.
- "Actions Data Stream" - or you can just fix your logging.
There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.
[1] - https://github.com/orgs/community/discussions/160682#discuss...
reply