Nothing screams panic like accidentally deploying an older, broken staging build, that also run outdated database migrations on start, straight to production

We get an email from the customer that a feature they “are using every day” is broken and it needs to be fixed ASAP. Sure, seems to be broken in current build. Run git blame to see when it stopped working. Last change may of last year. It’s been broken for almost a year and it took them until now to notice even though they “use it every day”.

Dear Build Server, my code is fine, it works locally, we’re all happy. Then I send it to you and you shit the bed.

If you were were a person you’d be the one sitting in a corner at parties, crying to get attention.

(For anyone feeling the need to state the obvious, I’m fully aware I’ve broken something, I just want to be grumpy about it)

Why the fuck is the master almost constantly broken? And not even "some feature I'm working on doesn't work"-broken but "can't build this shit"-broken. What the fuck is the workflow here that it's apparently acceptable? I wasn't able to do SHIT today because of it. Almost whole fucking day wasted.

There was a boom and my computer was dead. No power to the motherboard at all. Strong burning stench.

And I have no spare parts to test if the motherboard or the PSU is broken... My money is on the PSU. No visible marks anywhere. But could be both.

It has been roughly ten years... GPU was updated. But besides that, same computer.

Let's see, best I order a new PSU and see if it works and if it doesn't I salvage the GPU and build a new computer around it. But hey, that sucks.

When the CTO/CEO of your "startup" is always AFK and it takes weeks to get anything approved by them (or even secure a meeting with them) and they have almost-exclusive access to production and the admin account for all third party services.

Want to create a new messaging channel? Too bad! What about a new repository for that cool idea you had, or that new microservice you're expected to build. Expect to be blocked for at least a week.

When they also hold themselves solely responsible for security and operations, they've built their own proprietary framework that handles all the authentication, database models and microservice communications.

Speaking of which, there's more than six microservices per developer!

Oh there's a bug or limitation in the framework? Too bad. It's a black box that nobody else in the company can touch. Good luck with the two week lead time on getting anything changed there. Oh and there's no dedicated issue tracker. Have you heard of email?

When the systems and processes in place were designed for "consistency" and "scalability" in mind you can be certain that everything is consistently broken at scale. Each microservice offers:

1. Anemic & non-idempotent CRUD APIs (Can't believe it's not a Database Table™) because the consumer should do all the work.
2. Race Conditions, because transactions are "not portable" (but not to worry, all the code is written as if it were running single threaded on a single machine).
3. Fault Intolerance, just a single failure in a chain of layered microservice calls will leave the requested operation in a partially applied and corrupted state. Ger ready for manual intervention.
4. Completely Redundant Documentation, our web documentation is automatically generated and is always of the form //[FieldName] of the [ObjectName].
5. Happy Path Support, only the intended use cases and fields work, we added a bunch of others because YouAreGoingToNeedIt™ but it won't work when you do need it. The only record of this happy path is the code itself.

Consider this, you're been building a new microservice, you've carefully followed all the unwritten highly specific technical implementation standards enforced by the CTO/CEO (that your aware of). You've decided to write some unit tests, well um.. didn't you know? There's nothing scalable and consistent about running the system locally! That's not built-in to the framework. So just use curl to test your service whilst it is deployed or connected to the development environment. Then you can open a PR and once it has been approved it will be included in the next full deployment (at least a week later).

Most new 'services' feel like the are about one to five days of writing straightforward code followed by weeks to months of integration hell, testing and blocked dependencies.

When confronted/advised about these issues the response from the CTO/CEO
varies:

(A) "yes but it's an edge case, the cloud is highly available and reliable, our software doesn't crash frequently".
(B) "yes, that's why I'm thinking about adding [idempotency] to the framework to address that when I'm not so busy" two weeks go by...
(C) "yes, but we are still doing better than all of our competitors".
(D) "oh, but you can just [highly specific sequence of undocumented steps, that probably won't work when you try it].
(E) "yes, let's setup a meeting to go through this in more detail" *doesn't show up to the meeting*.
(F) "oh, but our customers are really happy with our level of [Documentation]".

Sometimes it can feel like a bit of a cult, as all of the project managers (and some of the developers) see the CTO/CEO as a sort of 'programming god' because they are never blocked on anything they work on, they're able to bypass all the limitations and obstacles they've placed in front of the 'ordinary' developers.

There's been several instances where the CTO/CEO will suddenly make widespread changes to the codebase (to enforce some 'standard') without having to go through the same review process as everybody else, these changes will usually break something like the automatic build process or something in the dev environment and its up to the developers to pick up the pieces. I think developers find it intimidating to identify issues in the CTO/CEO's code because it's implicitly defined due to their status as the "gold standard".

It's certainly frustrating but I hope this story serves as a bit of a foil to those who wish they had a more technical CTO/CEO in their organisation. Does anybody else have a similar experience or is this situation an absolute one of a kind?

A system to build note-taking systems. tatatap dot com.

It’s the most successful for a few reasons: it got launched, people find it useful, but most importantly it’s been fun and continues to be fun to work on.

I think the fun-to-make factor is massively underestimated as a success indicator. Working on the right product (whatever that means) that is unenjoyable is like using an amazing computer with a broken keyboard. It’s never going to work.

Sure, with any project there’s annoying stuff, but it’s the trend overall. Is the core functionality fun to work on?

In the case of Tap the core component is a notation parser, open sourced called sowhat, github dot com/tatatap-com/sowhat

That was super fun to make and learn about lexing and parsing. It’s pretty far along but there’s still a lot I’m planning to add.