Oh, man, I just realized I haven't ranted one of my best stories on here!

So, here goes!

A few years back the company I work for was contacted by an older client regarding a new project.

The guy was now pitching to build the website for the Parliament of another country (not gonna name it, NDAs and stuff), and was planning on outsourcing the development, as he had no team and he was only aiming on taking care of the client service/project management side of the project.

Out of principle (and also to preserve our mental integrity), we have purposely avoided working with government bodies of any kind, in any country, but he was a friend of our CEO and pleaded until we singed on board.

Now, the project itself was way bigger than we expected, as the wanted more of an internal CRM, centralized document archive, event management, internal planning, multiple interfaced, role based access restricted monster of an administration interface, complete with regular user website, also packed with all kind of features, dashboards and so on.

Long story short, a lot bigger than what we were expecting based on the initial brief.

The development period was hell. New features were coming in on a weekly basis. Already implemented functionality was constantly being changed or redefined. No requests we ever made about clarifications and/or materials or information were ever answered on time.

They also somehow bullied the guy that brought us the project into also including the data migration from the old website into the new one we were building and we somehow ended up having to extract meaningful, formatted, sanitized content parsing static HTML files and connecting them to download-able files (almost every page in the old website had files available to download) we needed to also include in a sane way.

Now, don't think the files were simple URL paths we can trace to a folder/file path, oh no!!! The links were some form of hash combination that had to be exploded and tested against some king of database relationship tables that only had hashed indexes relating to other tables, that also only had hashed indexes relating to some other tables that kept a database of the website pages HTML file naming. So what we had to do is identify the files based on a combination of hashed indexes and re-hashed HTML file names that in the end would give us a filename for a real file that we had to then search for inside a list of over 20 folders not related to one another.

So we did this. Created a script that processed the hell out of over 10000 HTML files, database entries and files and re-indexed and re-named all this shit into a meaningful database of sane data and well organized files.

So, with this we were nearing the finish line for the project, which by now exceeded the estimated time by over to times.

We test everything, retest it all again for good measure, pack everything up for deployment, simulate on a staging environment, give the final client access to the staging version, get them to accept that all requirements are met, finish writing the documentation for the codebase, write detailed deployment procedure, include some automation and testing tools also for good measure, recommend production setup, hardware specs, software versions, server side optimization like caching, load balancing and all that we could think would ever be useful, all with more documentation and instructions.

As the project was built on PHP/MySQL (as requested), we recommended a Linux environment for production. Oh, I forgot to tell you that over the development period they kept asking us to also include steps for Windows procedures along with our regular documentation. Was a bit strange, but we added it in there just so we can finish and close the damn project.

So, we send them all the above and go get drunk as fuck in celebration of getting rid of them once and for all...

Next day: hung over, I get to the office, open my laptop and see on new email. I only had the one new mail, so I open it to see what it's about.

Lo and behold! The fuckers over in the other country that called themselves "IT guys", and were the ones making all the changes and additions to our requirements, were not capable enough to follow step by step instructions in order to deploy the project on their servers!!!

[Continues in the comments]

Client: Can you provide some kind of guaranteed timeline that you're going to be able to move our website to our new servers with the optimizations implemented? I know you said it should take a week, but we have 3 weeks to get this moved over and we cannot afford to be double billed. I'm waiting to fire up the new server until you can confirm.

Me: As I said, it SHOULD take about a week, but that's factoring in ONLY the modifications being made for optimization and a QA call to review the website. This does not account for your hosting provider needing to spin up a new server.

We also never offered to move your website over to said new server. I sent detailed instructions for your provider to move a copy of the entire website over and have it configured and ready to point your domain over to, in order to save time and money since your provider won't give us the access necessary to perform a server-to-server transfer. If you are implying that I need to move the website over myself, you will be billed for that migration, however long it takes.

Client: So you're telling me that we paid $950 for 10 hours of work and that DOESN'T include making the changes live?

Me: Why would you think that the 10 hours that we're logged for the process of optimizing your website include additional time that has not been measured? When you build out a custom product for a customer, do you eat the shipping charges to deliver it? That is a rhetorical question of course, because I know you charge for shipping as well. My point is that we charge for delivery just as you do, because it requires our time and manpower.

All of this could have been avoided, but you are the one that enforced the strict requirement that we cannot take the website down for even 1 hour during off-peak times to incorporate the changes we made on our testbed, so we're having to go through this circus in order to deliver the work we performed.

I'm not going to give you a guarantee of any kind because there are too many factors that are not within our control, and we're not going to trap ourselves so you have a scapegoat to throw under the bus if your boss looks to you for accountability. I will reiterate that we estimate it would take about a week to implement, test and run through a full QA together, as we have other clients within our queue and our time must be appropriately blocked out each day. However, the longer you take to pull the trigger on this new server, the longer it will take on my end to get the work scheduled within the queue.

Client: If we get double billed, we're taking that out of what we have remaining to pay you.

Me: On the subject of paying us, you signed a contract acknowledging that you would pay us the remaining 50% after you approved the changes, which you did last week, in order for us to deliver the project. Thank you for the reminder that your remaining balance has not yet been paid. I'll have our CFO resend the invoice for you to remit payment before we proceed any further.

---

I love it when clients give me shit. I just give it right back.

Worst thing you've seen another dev do? Long one, but has a happy ending.

Classic 'Dev deploys to production at 5:00PM on a Friday, and goes home.' story.

The web department was managed under the the Marketing department, so they were not required to adhere to any type of coding standards and for months we fought with them on logging. Pre-Splunk, we rolled our own logging/alerting solution and they hated being the #1 reason for phone calls/texts/emails every night.

Wanting to "get it done", 'Tony' decided to bypass the default logging and send himself an email if an exception occurred in his code.

At 5:00PM on a Friday, deploys, goes home.

Around 11:00AM on Sunday (a lot folks are still in church at this time), the VP of IS gets a call from the CEO (who does not go to church) about unable to log into his email. VP has to leave church..drive home and find out he cannot remote access the exchange server. He starts making other phone calls..forcing the entire networking department to drive in and get email back up (you can imagine not a group of happy people)

After some network-admin voodoo, by 12:00, they discover/fix the issue (know it was Tony's email that was the problem)

We find out Monday that not only did Tony deploy at 5:00 on a Friday, the deployment wasn't approved, had features no one asked for, wasn't checked into version control, and the exception during checkout cost the company over $50,000 in lost sales.

Was Tony fired? Noooo. The web is our cash cow and Tony was considered a top web developer (and he knew that), Tony decided to blame logging. While in the discovery meeting, Tony told the bosses that it wasn't his fault logging was so buggy and caused so many phone calls/texts/emails every night, if he had been trained properly, this problem could have been avoided.

Well, since I was responsible for logging, I was next in the hot seat.

For almost 30 minutes I listened to every terrible thing I had done to Tony ever since he started. I was a terrible mentor, I was mean, I was degrading, etc..etc.
Me: "Where is this coming from? I barely know Tony. We're not even in the same building. I met him once when he started, maybe saw him a couple of times in meetings."
Andrew: "Aren't you responsible for this logging fiasco?"
Me: "Good Lord no, why am I here?"
Andrew: "I'll rephrase so you'll understand, aren't you are responsible for the proper training of how developers log errors in their code? This disaster is clearly a consequence of your failure. What do you have to say for yourself?"
Me: "Nothing. Developers are responsible for their own choices. Tony made the choice to bypass our logging and send errors to himself, causing Exchange to lockup and losing sales."
Andrew: "A choice he made because he was not properly informed of the consequences? Again, that is a failure in the proper use of logging, and why you are here."
Me: "I'm done with this. Does John know I'm in here? How about you get John and you talk to him like that."
'John' was the department head at the time.
Andrew:"John, have you spoken to Tony?"
John: "Yes, and I'm very sorry and very disappointed. This won't happen again."
Me: "Um...What?"
John: "You know what. Did you even fucking talk to Tony? You just sit in your ivory tower and think your actions don't matter?"
Me: "Whoa!! What are you talking about!? My responsibility for logging stops with the work instructions. After that if Tony decides to do something else, that is on him."
John: "That is not how Tony tells it. He said he's been struggling with your logging system everyday since he's started and you've done nothing to help. This behavior ends today. We're a fucking team. Get off your damn high horse and help the little guy every once in a while."
Me: "I don't know what Tony has been telling you, but I barely know the guy. If he has been having trouble with the one line of code to log, this is the first I've heard of it."
John: "Like I said, this ends today. You are going to come up with a proper training class and learn to get out and talk to other people."

Over the next couple of weeks I become a powerpoint wizard and 'train' anyone/everyone on the proper use of logging. The one line of code to log. One line of code.

A friend 'Scott' sits close to Tony (I mean I do get out and know people) told me that Tony poured out the crocodile tears. Like cried and cried, apologizing, calling me everything but a kitchen sink,...etc. It was so bad, his manager 'Sally' was crying, her boss 'Andrew', was red in the face, when 'John' heard 'Sally' was crying, you can imagine the high levels of alpha-male 'gotta look like I'm protecting the females' hormones flowing.

Took almost another year, Tony released a change on a Friday, went home, web site crashed (losses were in the thousands of $ per minute this time), and Tony was not let back into the building on Monday (one of the best days of my life).

rant? rant!

I work for a company that develops a variety of software solutions for companies of varying sizes. The company has three people in charge, and small teams that each worked on a certain project. 9 months ago I joined the company as a junior developer, and coincidentally, we also started working on our biggest project so far - an online platform for buying groceries from a variety of vendors/merchants and having them be delivered to your doorstep on the same day (hadn't been done to this scale in Estonia yet). One of the people from management joined the team working on that. The company that ordered this is coincidentally being run by one of the richest men in Estonia. The platform included both the actual website for customers to use, a logistics system for routing between the merchants, the warehouse, and the customers, as well as a bunch of mobile apps for the couriers, warehouse personnel, etc. It was built on Node.js with Hapi (for the backend stuff), Angular 2 (for all the UIs, including the apps which are run through a WebView wrapper), and PostgreSQL (for the database). The deadline for the MVP we (read: the management) gave them, but we finished it in about 7 months in a team of five.

The hours were insane, from 10 AM to 10 PM if lucky. When we weren't lucky (which was half of the time, if not more), we had to work until anywhere from 12 PM to 3 AM, sometimes even the whole night. The weekends weren't any better, for the majority of the time we had to put in even more extra hours on the weekends. Luckily, we were paid extra for them, but the salary was no way near fair (the majority of the team earned about 1000€/mo after taxes in a country where junior developers usually earn 1500€/month). Also because of the short deadline given to us, we skipped all the important parts like writing tests, doing CI, code reviews, feature branching/PR's, etc. I tried pushing the team and the management to at least write tests and make feature branches/PRs, but the management always told me that there wasn't enough time to coordinate and work on all that, that we'll do that after launching the MVP, etc. We basically just wrote features, tested them by hand, and pushed into the "test" branch which would later get tested and merged into master.

During development, one of the other juniors managed to write the worst kind of Angular code you could imagine - enormous amounts of duplication, no reusable components (every view contained the everything used in the view, so popups and other parts that should logically be reusable were in every view separately), fuck - even the HTML was broken (the most memorable for me were the "table > tr > div > td" ones, but that's barely scratching the surface). He left a few months into the project, and we had to build upon his shit, ever so slightly trying to fix the shit he produced. This could have definitely been avoided if we did code reviews.

A month after launching the MVP for internal testing, the guy working on the logistics system had burned out and left the company (he's earning more than twice the salary he got here, happy for him, he is a great coder and an even better team player). This could have been avoided if this project had been planned better, but I can't really blame them, since it was the first project they had at this scale (even though they had given longer deadlines for projects way smaller than this).

After we finished and launched the MVP, the second guy from management joined, because he saw we needed extra help. Again I tried to push us into investing the time to write tests for the system (because at this point we had created an unstable cluster fuck of a codebase), but again to no avail. The same "no time, just test it manually for now, we'll do that later when we have time" bullshit from management.

Now, a few weeks ago, the third guy from management joined. He saw what a disaster our whole project was. Him joining was simply a blessing from the skies. He started off by writing migrations using sequelize. I talked to him about writing tests and everything, and he actually listened. He told me that I'm gonna be the one writing them, and also talked to the rest of management about it. I was overjoyed. I could actually hear the bitterness in the voices of the rest of management when they told me how to write the tests, what to test, etc. But I didn't give a flying rat's ass, I was hapi.

I was told to start off by writing a smoke test for the whole client flow using Puppeteer. I got even happier, since I was finally able to again learn new things (this stopped at about 4 or 5 months into the project).

I'm using jest as the framework and started writing the tests in TypeScript. Later I found a library called jest-extended, but it didn't have type defs, so I decided to write them and, for the first time in my life, contribute to the open source community.

So today was shite. I get done with classes for today after going to work in the morning( college student) and I get a phone call. SURPRISE! We have a client meeting in twenty minutes.

My class runs from 5-9 and the meeting was scheduled at 7:30. I was let out early thank god. Then made sure that we had deliverables for the project running and ready to go.

Meeting with the client went well and he was pleased with my progress.

Now I have been the only one developing any sort of deliverable for this website. It originally started out with three people. The main developer will call him Cunt and a front-end dev that isn't on the project anymore.

FUCKING CUNT DECIDED TO COME AT ME FOR SAYING THAT THERE WAS A MISCOMMUNICATION ERROR AND ONE OF THE TASKS WASNT FINISHED. THEN CUNT ASKED IF I HAD ALL OF THE FRONT DEVS PAGES TO WHICH I SIGHED YES. CUNT FUCKFACE PROCEEDED TO EDUCATE ME ON HOW TO TALK TO OTHER DEVS AND NOT MAKE CRITICIZE HER CODE. UM CUNT YOU HAVENT COMMUNICATED WITH HER FOR THE THREE MONTHS IVE BEEN ON THIS PROJECT. AND YOUVE CONSTANTLY AVOIDED BEING IN THE OFFICE WHEN ANYONE IS PRESENT AND EVEN SHOOED HER AWAY. THE FACT THAT CUNT ISNT FIRED IS BECAUSE I TOOK LEAD ON HIS PROJECT AND HAVE MAKING EVERY DEADLINE FOR THIS AND OTHER PROJECTS.

*breaths*

Moral of this. Don't be a CUNT!

Being the only dev in a media company, I believe my coworkers looked upon me as the evil Internet reincarnated into a living person.

Most of them avoided me like a leprous, and I experienced long periods being almost isolated.

I did almost quit so many times I've lost count.

I eventually found another job, where there's lots of love, caring and we all embrace each other's skills. It feels like I'm a butterfly who finally got to break out of my miserable cocoon.

Did a bunch more cowboy coding today as I call it (coding in vi on production). Gather 'round kiddies, uncle Logan's got a story fer ya…

First things first, disclaimer: I'm no sysadmin. I respect sysadmins and the work they do, but I'm the first to admit my strengths definitely lie more in writing programs rather than running servers.

Anyhow, I recently inherited someone else's codebase (the story of my profession career, but I digress) and let me tell you this thing has amateur hour written all over it. It's written in PHP and JavaScript by a self-taught programmer who apparently discovered procedural programming and decided there was nothing left to learn and stopped there (no disrespect to self-taught programmers).

I could rant for days about the various problems this codebase has, but today I have a very specific story to tell. A story about errors and logs.

And it all started when I noticed the disk space on our server was gradually decreasing.

So today I logged onto our API server (Ubuntu running Apache/PHP) and did a df -h to check the disk space, and was surprised to see that it had noticeably decreased since the last time I'd checked when everything was running smoothly. But seeing as this server does not store any persistent customer data (we have a separate db server) and purely hosts the stateless API, it should NOT be consuming disk space over time at all.

The only thing I could think of was the logs, but the logs were very quiet, just the odd benign message that was fully expected. Just to be sure I did an ls -Sh to check the size of the logs, and while some of them were a little big, nothing over a few megs. Nothing to account for gigabytes of disk space gradually disappearing.

What could it be? I wondered.

cd ../..
du . | sort --sort=numeric

What's this? 2671132 K in some log folder buried in the api source code? I cd into it and it turns out there are separate PHP log files in there, split up by customer, so that each customer of ours (we have 120) has their own respective error log! (Why??)

Armed with this newfound piece of (still rather unbelievable) evidence I perform a mad scramble to search the codebase for where this extra logging is happening and sure enough I find a custom PHP error handler that is capturing (most) errors and redirecting them to these individualized log files.

Conveniently enough, not ALL errors were being absorbed though, so I still knew the main error_log was working (and any time I explicitly error_logged it would go there, so I was none the wiser that this other error-catching was even happening).

Needless to say I removed the code as quickly as I found it, tail -f'd the error_log and to my dismay it was being absolutely flooded with syntax errors, runtime PHP exceptions, warnings galore, and all sorts of other things.

My jaw almost hit the floor. I've been with this company for 6 months and had no idea these errors were even happening!

The sad thing was how easy to fix all the errors ended up being. Most of them were "undefined index" errors that could have been completely avoided with a simple isset() check, but instead ended up throwing an exception, nullifying any code that came after it.

Anyway kids, the moral of the story is don't split up your log files. It makes absolutely no sense and can end up obscuring easily fixable bugs for half a year or more!

Happy coding.

Not a Story about an actual hack, but a story about people being dumb and using hacks as an excuse.

A few weeks ago my little cousin would reach out to me because "his Account was hacked...". Supposedly his League of Legends account was hacked by a guy of his own age (14) and this guy was boasting about it.

So i asked the usual things: "Has the email account been hijacked? Did anyone know about details to your acvount access? Etc..."

Turns out that one if his "friends" knew his password and username, but suppsedly erased these Informationen. And that was the part i didn't buy.

This was the point where he lost. Just because i am a programmer does not mean i can retrieve an account he lost because of a dumb mistake that could have easily been avoided. And that guy who was boasting about hacking LoL Account was coincidentally freinds with the friend who had the user credentials and password.

Moral of the Story? The biggest security weakness is almost always the user or a human in between...

Rant considering the latest Cyber attack and the news around it.

(A recap: a lot of Windows computers were infected with ransomware (due to security hole on Windows), which demanded 300$ in bitcoins to unlock data. After 3 days the price would double, and after 7 days the data was to be deleted)
1) In our country, one of the biggest companies was attacked (car factory). The production stopped and they got for around 1 000 000€ damage in less than 24h (1300 people without work). The news said that they were attacked because they are such a big company and were charged more, as the hackers "knew who they were dealing with" - another reason being the fact that the text was in croatian (which is our neighbor country), but noone realized that it is just a simple google translate of english text - which is obviously not true. The hackers neither know nor care who is hacked, and will charge everyone the same. They only care about the payment.
2) In UK whole (or large part) of medical infrastructure went down. The main thing everyone was saying was: "Nobody's data is stolen". Which, again, is obvious. But noone said anything about data being deleted after a week, which includes pretty much whole electronic medical record of everyone and is pretty serious.
And by the way, the base of the ransomware is code which was stolen from NSA.
All that millions and millions of dollars of damage could be avoided by simply paying the small fee.
The only thing that is good is that (hopefully) the people will learn the importance of backups. And opening weird emails.
P.S. I fucking hate all that 'hacky thingys' they have all over the news.

I've spent a lot of time messing around with C, having struggled with object-oriented programming (due to not really knowing how best to structure things, not knowing when to apply certain design patterns).

When writing C code, I'd write OOP-esque code (pass around a struct to routines to do things with it) and enjoyed just making things happen without having to think too much about the overall design. But then I'd crave being able to use namespaces, and think about how the code would be tidier if I used exceptions instead of having every routine return an error code...

Working with Python and Node over the past couple of years has allowed me to easily get into OOP (no separate declaration/definition, loose typing etc.) and from that I've made some fairly good design decisions. I'd implemented a few design patterns without even realising which patterns they were - later reading up on them and thinking "hey, that's what I used earlier!"

I've also had a bit of an obsession with small executable files - using templates and other features of C++ add some bloat (on Windows at least) compared to C. There were other gripes I had with C++, mostly to do with making things modular (dynamic linking etc.) but really it's irrelevant/unreasonable.

And yes, for someone who doesn't like code bloat, working with Node is somewhat ironic... (hello, node_modules...)

So today I decided to revisit C++ and dust off my old copy of C++ in a Nutshell, and try to see if I could write some code to do things that I struggled with before. One nice thing is that this book was printed in 2003, yet all of its content is still relevant. Of course, there are newer C++ standards, but I can happily just hack away and avoid using anything that has been deprecated.

One thing I've always avoided is dynamic_cast because every time I read about it, I read that "it's slow". So I just tried to work around it when really if it's the right tool for the job, I might as well use it... It's really useful!

Anyway, now I've typed all this positivity about C++ I will probably find a little later on that I hit a wall with what I'm doing and give up again... :p

Moving files is emotionally easier than copying and deleting files, and moving eliminates the risk of selecting the wrong files at the deletion part.

I have read that it is safer to manually copy and manually delete files rather than to move it, but copying and deleting has a hidden risk that was not mentioned: selecting the wrong files for deletion.

Moving files feels like moving an obstacle from one room to another. The deletion part of copying and deleting feels like destroying something, which is an added emotional barrier.

Technically, copying and deleting is safer, since there is no risk of source files being deleted without having been transferred as a result of a device disconnecting or the buggy media transfer protocol (MTP) failing to load the entire file list. However, on mass storage devices, this pretty much never happened to me, and on MTP, data loss can be avoided by not moving folders but opening the source folders and selecting all files and moving those out. This prevents a parent folder with incompletely loaded file listing from being deleted.

However, something that is not considered about copying and deleting is that the risk of selecting the wrong files in the deletion step exists. One might end up selecting files that were never copied.

Not only is moving straightforward and time-saving, but it has no emotional barrier and the risk of selecting the wrong files to delete from the source is eliminated, since a proper file manager like Nemo or Windows Explorer (mass storage only, not MTP) only deletes a moved file from the source after it has been properly transferred. The user does not need to pay attention to select the correct files to delete, since the file manager already did it.

How do you deal when people ignore you avoid you but didn't understood you??? I explained over and over and over again!!!!!!!!!!!!!!!! 😤😤😤😤😖😖

What's wrong with Stack Overflow? Honestly, somebody asked how to do something that should in most practical cases be avoided. I provided an answer and here comes the downvote army for no reason. I explicitly said it should be avoided but for the sake of experiment I posted the solution because I think people should explore what they can do with the language instead of feeling constricted to a set of standard recipes.
I don't buy into claims that this irrational elitist moderation is necessary for SO to be useful.
In the end, even their search sucks and most of us find it easier to search SO using Google instead of their native search.
I remember when I was a student at a programme which admitted both people with linguistic and computational background how hard it was for the linguists to even start writing code and I would always try to help them and relieve the frustration.
For me, it took years to start writing a high quality code and more than 6 years to become productive while writing quality code.
Do we forget we all began somewhere? I honestly don't care about building an immense "objective" problem solving tool for someone else to earn money at the expense of treating people the way SO community does.
I think it would be way better if SO managed to distribute questions in a more relevant manner and stopped holding onto their "objectivism", which is in itself a questionable concept.
Even simply separating questions into how popular they are could move the useful ones forward without radically cancelling and hurting new people.
I like to see people thinking differently and see their questions reveal what they know and what they don't. There's nothing wrong with pointing people to already answered questions, correcting them etc. And I get that there are many people being annoying when asking, but I never forget there is a person on the other side and I would never want to destroy their potential just to massage my ego and "reputation". And heck who cares about their reputation? Show your Github, CV, talk smart in an interview and you'll get the job. And in the end, wouldn't you feel greater inner joy from helping a person grow instead of seeing only your reputation?

Being told to continuously restart a server over the weekend while someone's API script kept crashing the server. Yes, they knew it was doing it. No, they "claimed" they didn't know how to slow down the amount of new API sessions. I avoided all their requests after that.

Best:
Seeing ALL the members of my team finally coming into their own. One person tackled our entire not-at-all-simple CI/CD setup from scratch knowing nothing about any of it and, while not without bumps in the road, did an excellent job overall (and then did the same for some other projects since he found himself being the SME). Two of my more junior people took on some difficult tasks that required them to design and build some tricky features from the ground-up, rather than me giving them a ton of guidance, design and even a start on the basic code early on (I just gave them some general descriptions of what I was looking for and then let them run with it). Again, not without some hiccups, but they ultimately delivered and learned a lot in the process and, I think, gained a new sense of self-confidence, which to me is the real win. And my other person handled some tricky high-level stuff that got him deep in the weeds of all the corporate procedures I'd normally shield them all from and did very well with it (and like the other person, wound up being an SME and doing it for some other projects after that). It took a while to get here, but I finally feel like I don't need to do all the really difficult stuff myself, I can count on them now, and they, I think, no longer feel like they're in over their heads if I throw something difficult at them.

Worst:
A few critical bugs slipped into production this year, with a few requiring some after-hours heroics to deal with (and, unfortunately, due to the timing, it all fell on me). Of course, that just tells us that next year we really need to focus on more robust automated testing (though, in reality, at least one of the issues almost certainly would not - COULD NOT - have been caught before-hand anyway, and that's probably true for more than just one of them). We had avoided major issues the previous three years we've been live, so this was unusual. Then again, it's in a way a symptom of success because with more users and more usage, both of which exploded this year, typically does come more issues discovered, so I guess it tempers the bad just a little bit.

Recently we noticed a part of our web application wasn't working. After some hours of looking into it (it's an old, convoluted application), it became clear another part of the application timed out trying to get a connection from the db connection pool.

We call db admins, they respond "oh yeah looks like the DB CPUs are at 100% load. I'll do something about it." and a short while later everything was working. So now I think, our hours of looking into it and a lot of people not being able to work could have been avoided if the DB admins had some form of alerting. But also we could improve our monitoring too, had we tracked calls made to our DB.

Question: Do you think I should call the DB guys, telling them they need alerting, or should I add tracing/monitoring around our DB calls, or both? Do you think I should consider any additional actions I haven't thought of?