Yesterday: Senior dev messages out a screenshot of someone using an extension method I wrote (he didn’t know I wrote it)..

SeniorDev: “OMG…that has to be the stupidest thing I ever saw.”
Me: “Stupid? Why?”
SeniorDev: “Why are they having to check the value from the database to see if it’s DBNull and if it is, return null. The database value is already null. So stupid.”
Me: “DBNull is not null, it has a value. When you call the .ToString, it returns an empty string.”
SeniorDev: ”No it doesn’t, it returns null.”
<oh no he didn’t….the smack down begins>
Me: “Really? Are you sure?”
SeniorDev: “Yes! And if the developer bothered to write any unit tests, he would have known.”
Me: “Unit tests? Why do you assume there aren’t any unit tests? Did you look?”
<at this moment, couple other devs take off their head phones and turn around>
SeniorDev:”Well…uh…I just assumed there aren’t because this is an obvious use case. If there was a test, it would have failed.”
Me: “Well, let’s take a look..”
<open up the test project…navigate to the specific use case>
Me: “Yep, there it is. DBNull.Value.ToString does not return a Null value.”
SeniorDev: “Huh? Must be a new feature of C#. Anyway, if the developers wrote their code correctly, they wouldn’t have to use those extension methods. It’s a mess.”
<trying really hard not drop the F-Bomb or two>
Me: “Couple of years ago the DBAs changed the data access standard so any nullable values would always default to null. So no empty strings, zeros, negative values to indicate a non-value. Downside was now the developers couldn’t assume the value returned the expected data type. What they ended up writing was a lot of code to check the value if it was DBNull. Lots of variations of ‘if …’ , ternary operators, some creative lamda expressions, which led to unexpected behavior in the user interface. Developers blamed the DBAs, DBAs blamed the developers. Remember, Tom and DBA-Sam almost got into a fist fight over it.”
SeniorDev: “Oh…yea…but that’s a management problem, not a programming problem.”
Me: “Probably, but since the developers starting using the extension methods, bug tickets related to mis-matched data has nearly disappeared. When was the last time you saw DBA-Sam complain about the developers?”
SeniorDev: “I guess not for a while, but it’s still no excuse.”
Me: “Excuse? Excuse for what?”
<couple of awkward seconds of silence>
SeniorDev: “Hey, did you guys see the video of the guy punching the kangaroo? It’s hilarious…here, check this out.. ”

Pin shoulders the mat…1 2 3….I win.

After listening to two of our senior devs play ping pong with a new member of our team for TWO DAYS!
DevA: "Try this.."
Junior: "Didn't work"
DevB: "Try that .."
Junior: "Still not working"

I ask..
Me:"What is the problem?"
Few ums...uhs..awkward seconds of silence

Junior: "App is really slow. Takes several seconds to launch and searching either crashes or takes a really long time."
DevA: "We've isolated the issue with Entity Framework. That application was written back when we used VS2010. Since that application isn't used very often, no one has had to update it since."
DevB: "Weird part is the app takes up over 3 gigs of ram. Its obviously a caching issue. We might have to open up a ticket with Microsoft."
Me: "Or remove EF and use ADO."
DevB: "That would be way too much work. The app is supposed to be fully deprecated and replaced this year."
Me: "Three of you for the past two days seems like a lot of work. If EF is the problem, you remove EF."
DevA: "The solution is way too complicated for that. There are 5 projects and 3 of those have circular dependencies. Its a mess."
DevB: "No fracking kidding...if it were written correctly the first time. There aren't even any fracking tests."
Me:"Pretty sure there are only two tables involved, maybe 3 stored procedures. A simple CRUD app like this should be fairly straight forward."
DevB: "Can't re-write the application, company won't allow it. A redesign of this magnitute could take months. If we can't fix the LINQ query, we'll going to have the DBAs change the structures to make the application faster. I don't see any other way."

Holy frack...he didn't just say that.

Over my lunch hour, I strip down the WPF application to the basics (too much to write about, but the included projects only had one or two files), and created an integration test for refactoring the data access to use ADO. After all the tests and EF removed, the app starts up instantly and searches are also instant. Didn't click through all the UI, but the basics worked.

Sat with Junior, pointed out my changes (the 'why' behind the 'what') ...and he how he could write unit tests around the ViewModel behavior in the UI (and making any changes to the data access as needed).

Today's standup:
Junior: "Employee app is fixed. Had some help removing Entity Framework and how it starts up fast and and searches are instant. Going to write unit tests today to verify the UI behaivor. I'll be able to deploy the application tomorrow."
DevA: "What?! No way! You did all that yesterday?"
Me: "I removed the Entity Framework over my lunch hour. Like I said, its basic CRUD and mostly in stored procedures. All the data points are covered by integration tests, but didn't have time for the unit tests. It's likely I broke some UI behavior, but the unit tests should catch those."
DevB: "I was going to do that today. I knew taking out Entity Framework wouldn't be a big deal."

Holy fracking frack. You fracking lying SOB. Deeeep breath...ahhh...thanks devRant. Flame thrower event diverted.

#3 Worst thing I've seen a co-worker do?

A 20-something dev, 'A', back in the early days of twitter+facebook would post all his extracurricular activities (drinking, partying, normal young-buck stuff). The dev mgr, 'J', at the time took offense because he felt 'A' was making the company look bad, so 'A' had a target on his back. Nothing 'A' did was good enough and, for example, 'J' had the source control czars review 'A's code to 'review' (aka = find anything wrong). Not sorting the 'using' statements, and extra line after the closing }, petty things like that. For those curious, orders followed+carried out by+led by 'T' in my previous rant.

As time went on and 'T' finding more and more 'wrong' with A's code, 'J' put A on disciplinary probation. 'A' had 90 days to turn himself around, or else.

A bright spot was 'A' was working on a Delphi -> C# conversion, so a lot of the code would be green-field development and by simply following the "standards", 'A' would be fine...so he thought.

About 2 weeks into the probation, 'A' was called into the J's office and berated because the conversion project was behind schedule, and if he didn't get the project back on track, 'A' wouldn't make it 30 days. I sat behind 'A' and he unloaded on me.
<'A' slams his phone on his desk>
Me: "Whoa...whats up?"
A: "Dude, I fucking hate this place, did you hear what they did?"
<I said no, then I think we spent an hour talking about it>
Me: "That all sucks. Don't worry about the code. Nobody cares what T thinks. Its not even your fault the project is behind, the DBAs are tasked with upgrades and it's not like anyone is waiting on you. It'll get done when it's done. Sounds like a witch hunt, what did you do? Be honest."
A: "Well, um...I kinda called out J, T, and those other assholes on facebook. I was drunk, pissed, and ...well...here we are."
Me: "Geez, what a bunch of whiney snowflakes. Keep your head down and you'll get thru it, or don't. Its not like you couldn't find another job tomorrow."
A: "This is my first job out of college and I don't want to disappoint my dad by quitting. I don't even know what I'm supposed to be doing. All J told me was to get better. What the fuk does that even mean?"
Me: "He didn't give you any goals? Crap, for someone who is a stickler for the rules, that's low, even for J."

Fast forward 2 weeks, I was attending MS TechEd and I was with another dev mgr, R.

R: "Did you hear? We had to let 'A' go today."
Me: "What the hell? Why?"
R: "He couldn't cut it, so we had to let him go."
Me: "Cut what? What did he do, specifically?"
R: "I don't know, 'A' was on probation, I guess he didn't meet the goals."
Me: "You guess? We fire a developer working on a major upgrade and you guess? What were these so-called goals?"
R: "Whoa...you're getting a little fire up. I don't know, maybe not adhering to coding standards, not meeting deadlines?"
Me: "OMG...we fire people for not forming code? Are you serious!?"
R: "Oh...yea...that does sound odd when you put it that way. I wish I'd talk to you before we left on this trip"
Me: "What?! You knew they were firing him *before* we left? How long did you know this was happening?"
R: "Honestly, for a while. 'A' really wasn't a team player."
Me: "That's dirty, the whole thing is dirty. We've done some shitty things to people, but this is low, even for J. The probation process is meant to improve, not be used as a witch hunt. I don't like that you stood around and let it happen. You know better."
R: "Yea, you're right, but doesn't change anything. J wanted to do it while most of us were at the conference in case 'A' caused a scene."
Me: "THAT MAKES IT WORSE! 'A' was blindsided and you knew it. He had no one there that could defend him or anything."
R: "Crap, crap, crap...oh crap...jeez...J had this planned all along...crap....there is nothing I can do no...its too late."
Me: "Yes there is. If 'A' comes to you for a letter of recommendation, you write one. If someone calls for reference, you give him a good one."
R: "Yea..yea...crap...I feel like shit...I need to go back to the room and lie down."

As the sun sets, it rises again. Within a couple of weeks, 'A' had another job at a local university. Within a year, he was the department manager, and now he is a vice president (last time I checked) of a college in Kansas City, MO.

Worst dev team failure I've experienced?

One of several.

Around 2012, a team of devs were tasked to convert a ASPX service to WCF that had one responsibility, returning product data (description, price, availability, etc...simple stuff)
No complex searching, just pass the ID, you get the response.

I was the original developer of the ASPX service, which API was an XML request and returned an XML response. The 'powers-that-be' decided anything XML was evil and had to be purged from the planet. If this thought bubble popped up over your head "Wait a sec...doesn't WCF transmit everything via SOAP, which is XML?", yes, but in their minds SOAP wasn't XML. That's not the worst WTF of this story.

The team, 3 developers, 2 DBAs, network administrators, several web developers, worked on the conversion for about 9 months using the Waterfall method (3~5 months was mostly in meetings and very basic prototyping) and using a test-first approach (their own flavor of TDD). The 'go live' day was to occur at 3:00AM and mandatory that nearly the entire department be on-sight (including the department VP) and available to help troubleshoot any system issues.

3:00AM - Teams start their deployments
3:05AM - Thousands and thousands of errors from all kinds of sources (web exceptions, database exceptions, server exceptions, etc), site goes down, teams roll everything back.

3:30AM - The primary developer remembered he made a last minute change to a stored procedure parameter that hadn't been pushed to production, which caused a side-affect across several layers of their stack.

4:00AM - The developer found his bug, but the manager decided it would be better if everyone went home and get a fresh look at the problem at 8:00AM (yes, he expected everyone to be back in the office at 8:00AM).

About a month later, the team scheduled another 3:00AM deployment (VP was present again), confident that introducing mocking into their testing pipeline would fix any database related errors.

3:00AM - Team starts their deployments.
3:30AM - No major errors, things seem to be going well. High fives, cheers..manager tells everyone to head home.
3:35AM - Site crashes, like white page, no response from the servers kind of crash. Resetting IIS on the servers works, but only for around 10 minutes or so.
4:00AM - Team rolls back, manager is clearly pissed at this point, "Nobody is going fucking home until we figure this out!!"
6:00AM - Diagnostics found the WCF client was causing the server to run out of resources, with a mix of clogging up server bandwidth, and a sprinkle of N+1 scaling problem. Manager lets everyone go home, but be back in the office at 8:00AM to develop a plan so this *never* happens again.

About 2 months later, a 'real' development+integration environment (previously, any+all integration tests were on the developer's machine) and the team scheduled a 6:00AM deployment, but at a much, much smaller scale with just the 3 development team members.

Why? Because the manager 'froze' changes to the ASPX service, the web team still needed various enhancements, so they bypassed the service (not using the ASPX service at all) and wrote their own SQL scripts that hit the database directly and utilized AppFabric/Velocity caching to allow the site to scale. There were only a couple client application using the ASPX service that needed to be converted, so deploying at 6:00AM gave everyone a couple of hours before users got into the office. Service deployed, worked like a champ.

A week later the VP schedules a celebration for the successful migration to WCF. Pizza, cake, the works. The 3 team members received awards (and a envelope, which probably equaled some $$$) and the entire team received a custom Benchmade pocket knife to remember this project's success. Myself and several others just stared at each other, not knowing what to say.

Later, my manager pulls several of us into a conference room
Me: "What the hell? This is one of the biggest failures I've been apart of. We got rewarded for thousands and thousands of dollars of wasted time."
<others expressed the same and expletive sediments>
Mgr: "I know..I know...but that's the story we have to stick with. If the company realizes what a fucking mess this is, we could all be fired."
Me: "What?!! All of us?!"
Mgr: "Well, shit rolls downhill. Dept-Mgr-John is ready to fire anyone he felt could make him look bad, which is why I pulled you guys in here. The other sheep out there will go along with anything he says and more than happy to throw you under the bus. Keep your head down until this blows over. Say nothing."

Devs: We need access to PROD DB in order to provide support you're asking us for.

Mgmt: No, we cannot trust you with PROD DB accesses. That DB contains live data and is too sensitive for you to fuck things up

Mgmt: We'll only grant PROD DB access to DBAs and app support guys

Mgmt: <hire newbies to app support>

App_supp: `update USER set invoice_directory = 54376; commit;`

----------------
I have nothing left to say....

Yesterday the web site started logging an exception “A task was canceled” when making a http call using the .Net HTTPClient class (site calling a REST service).

Emails back n’ forth ..blaming the database…blaming the network..then a senior web developer blamed the logging (the system I’m responsible for).

Under the hood, the logger is sending the exception data to another REST service (which sends emails, generates reports etc.) which I had to quickly re-direct the discussion because if we’re seeing the exception email, the logging didn’t cause the exception, it’s just reporting it. Felt a little sad having to explain it to other IT professionals, but everyone seemed to agree and focused on the server resources.

Last night I get a call about the exceptions occurring again in much larger numbers (from 100 to over 5,000 within a few minutes). I log in, add myself to the large skype group chat going on just to catch the same senior web developer say …
“Here is the APM data that shows logging is causing the http tasks to get canceled.”

FRACK!

Me: “No, that data just shows the logging http traffic of the exception. The exception is occurring before any logging is executed. The task is either being canceled due to a network time out or IIS is running out of threads. The web site is failing to execute the http call to the REST service.”

Several other devs, DBAs, and network admins agree.

The errors only lasted a couple of minutes (exactly 2 minutes, which seemed odd), so everyone agrees to dig into the data further in the morning.

This morning I login to my computer to discover the error(s) occurred again at 6:20AM and an email from the senior web developer saying we (my mgr, her mgr, network admins, DBAs, etc) need to discuss changes to the logging system to prevent this problem from negatively affecting the customer experience...blah blah blah.

FRACKing female dog!

Good news is we never had the meeting. When the senior web dev manager came in, he cancelled the meeting.

Turned out to be a hiccup in a domain controller causing the servers to lose their connection to each other for 2 minutes (1-minute timeout, 1 minute to fully re-sync). The exact two-minute burst of errors explained (and proven via wireshark).

People and their petty office politics piss me off.

Most satisfying bug I've fixed?

Fixed a n+1 issue with a web service retrieving price information. I initially wrote the service, but it was taken over by a couple of 'world class' monday-morning-quarterbacks.
The "Worst code I've ever seen" ... "I can't believe this crap compiles" types that never met anyone else's code that was any good.

After a few months (yes months) and heavy refactoring, the service still returned price information for a product. Pass the service a list of product numbers, service returns the price, availability, etc, that was it.

After a very proud and boisterous deployment, over the next couple of days the service seemed to get slower and slower. DBAs started to complain that the service was causing unusually high wait times, locks, and CPU spikes causing problems for other applications. The usual finger pointing began which ended up with "If PaperTrail had written the service 'correctly' the first time, we wouldn't be in this mess."

Only mattered that I initially wrote the service and no one seemed to care about the two geniuses that took months changing the code.

The dev manager was able to justify a complete re-write of the service using 'proper development methodologies' including budgeting devs, DBAs, server resources, etc..etc. with a projected year+ completion date.

My 'BS Meter' goes off, so I open up the code, maybe 5 minutes...tada...found it. The corresponding stored procedure accepts a list of product numbers and a price type (1=Retail, 2=Dealer, and so on). If you pass 0, the stored procedure returns all the prices.
Code basically looked like this..

public List<Prices> GetPrices(List<Product> products, int priceTypeId)
{
foreach (var item in products)
{
List<int> productIdsParameter = new List<int>();
productIdsParameter.Add(item.ProductID);
List<Price> prices = dataProvider.GetPrices(productIdsParameter, 0);
foreach (var price in prices)
{
if (price.PriceTypeID == priceTypeId)
{
prices = dataProvider.GetPrices(productIdsParameter, price.PriceTypeID);
return prices;
}
* Omitting the other 'WTF?' code to handle the zero price type
}
}
}

I removed the double stored procedure call, updated the method signature to only accept the list of product numbers (which it was before the 'major refactor'), deployed the service to dev (the issue was reproducible in our dev environment) and had the DBA monitor.

The two devs and the manager are grumbling and mocking the changes (they never looked, they assumed I wrote some threading monstrosity) then the DBA walks up..

DBA: "We're good. You hit the database pretty hard and the CPU never moved. Execution plans, locks, all good to go."
<dba starts to walk away>
DevMgr: "No fucking way! Putting that code in a thread wouldn't have fix it"
Me: "Um, I didn't use threads"
Dev1: "You had to. There was no way you made that code run faster without threads"
Dev2: "It runs fine in dev, but there is no way that level of threading will work in production with thousands of requests. I've got unit tests that prove our design is perfect."
Me: "I looked at what the code was doing and removed what it shouldn't be doing. That's it."
DBA: "If the database is happy with the changes, I'm happy. Good job. Get that service deployed tomorrow and lets move on"
Me: "You'll remove the recommendation for a complete re-write of the service?"
DevMgr: "Hell no! The re-write moves forward. This, whatever you did, changes nothing."
DBA: "Hell yes it does!! I've got too much on my plate already to play babysitter with you assholes. I'm done and no one on my team will waste any more time on this. Am I clear?"

Seeing the dev manager face turn red and the other two devs look completely dumbfounded was the most satisfying bug I've fixed.

Worst fight I've had with a co-worker?

Had my share of 'disagreements', but one that seemed like it could have gone to blows was a developer, 'T', that tried to man-splain me how ADO.Net worked with SQLServer.

<T walks into our work area>
T: "Your solution is going to cause a lot of problems in SQLServer"
Me: "No, its not, your solution is worse. For performance, its better to use ADO.Net connection pooling."
T: "NO! Every single transaction is atomic! SQLServer will prioritize the operation thread, making the whole transaction faster than what you're trying to do."
<T goes on and on about threads, made up nonsense about priority queues, on and on>
Me: "No it won't, unless you change something in the connection string, ADO.Net will utilize connection pooling and use the same SPID, even if you explicitly call Close() on the connection. You are just wasting code thinking that works."

T walks over, stands over me (he's about 6.5", 300+ pounds), maybe 6 inches away

T: "I've been doing .net development for over 10 years. I know what I'm doing!"

I turn my chair to face him, look up, cross my arms.

Me: "I know I'm kinda new to this, but let me show you something ..."

<I threw together a C# console app, simple connect, get some data, close the connection>

Me: "I'll fire up SQLProfiler and we can see the actual connection SPID and when sql server closes the SPID....see....the connection to SQLServer is still has an active SPID after I called Close. When I exit the application, SQLServer will drop the SPD....tada...see?"
T: "Wha...what is that...SQLProfiler? Is that some kind of hacking tool? DBAs should know about that!"
Me: "It's part of the SQLServer client tools, its on everyone's machine, including yours."
T: "Doesn't prove a damn thing! I'm going to do my own experiment and prove my solution works."
Me: "Look forward to seeing what you come up with ... and you haven't been doing .net for 10 years. I was part of the team that reviewed your resume when you were hired. You're going to have to try that on someone else."

About 10 seconds later I hear him from across the room slam his keyboard on his desk.

100% sure he would have kicked my ass, but that day I let him know his bully tactics worked on some, but wouldn't work on me.

Our web department was deploying a fairly large sales campaign (equivalent to a ‘Black Friday’ for us), and the day before, at 4:00PM, one of the devs emails us and asks “Hey, just a heads up, the main sales page takes almost 30 seconds to load. Any chance you could find out why? Thanks!”
We click the URL they sent, and sure enough, 30 seconds on the dot.
Our department manager almost fell out of his chair (a few ‘F’ bombs were thrown).
DBAs sit next door, so he shouts…
Mgr: ”Hey, did you know the new sales page is taking 30 seconds to open!?”
DBA: “Yea, but it’s not the database. Are you just now hearing about this? They have had performance problems for over week now. Our traces show it’s something on their end.”
Mgr: “-bleep- no!”
Mgr tries to get a hold of anyone …no one is answering the phone..so he leaves to find someone…anyone with authority.
4:15 he comes back..
Mgr: “-beep- All the web managers were in a meeting. I had to interrupt and ask if they knew about the performance problem.”
Me: “Oh crap. I assume they didn’t know or they wouldn’t be in a meeting.”
Mgr: “-bleep- no! No one knew. Apparently the only ones who knew were the 3 developers and the DBA!”
Me: “Uh…what exactly do they want us to do?”
Mgr: “The –bleep- if I know!”
Me: “Are there any load tests we could use for the staging servers? Maybe it’s only the developer servers.”
DBA: “No, just those 3 developers testing. They could reproduce the slowness on staging, so no need for the load tests.”
Mgr: “Oh my –bleep-ing God!”
4:30 ..one of the vice presidents comes into our area…
VP: “So, do we know what the problem is? John tells me you guys are fixing the problem.”
Mgr: “No, we just heard about the problem half hour ago. DBAs said the database side is fine and the traces look like the bottleneck is on web side of things.”
VP: “Hmm, no, John said the problem is the caching. Aren’t you responsible for that?”
Mgr: “Uh…um…yea, but I don’t think anyone knows what the problem is yet.”
VP: “Well, get the caching problem fixed as soon as possible. Our sales numbers this year hinge on the deployment tomorrow.”
- VP leaves -
Me: “I looked at the cache, it’s fine. Their traffic is barely a blip. How much do you want to bet they have a bug or a mistyped url in their javascript? A consistent 30 second load time is suspiciously indicative of a timeout somewhere.”
Mgr: “I was thinking the same thing. I’ll have networking run a trace.”

4:45 Networking run their trace, and sure enough, there was some relative path of ‘something’ pointing to a local resource not on development, it was waiting/timing out after 30 seconds. Fixed the path and page loaded instantaneously. Network admin walks over..
NetworkAdmin: “We had no idea they were having problems. If they told us last week, we could have identified the issue. Did anyone else think 30 second load time was a bit suspicious?”
4:50 VP walks in (“John” is the web team manager)..
VP: “John said the caching issue is fixed. Great job everyone.”
Mgr: “It wasn’t the caching, it was a mistyped resource or something in a javascript file.”
VP: “But the caching is fixed? Right? John said it was caching. Anyway, great job everyone. We’re going to have a great day tomorrow!”
VP leaves
NetworkAdmin: “Ouch…you feel that?”
Me: “Feel what?”
NetworkAdmin: “That bus John just threw us under.”
Mgr: “Yea, but I think John just saved 3 jobs. Remember that.”

The solution for this one isn't nearly as amusing as the journey.

I was working for one of the largest retailers in NA as an architect. Said retailer had over a thousand big box stores, IT maintenance budget of $200M/year. The kind of place that just reeks of waste and mismanagement at every level.

They had installed a system to distribute training and instructional videos to every store, as well as recorded daily broadcasts to all store employees as a way of reducing management time spend with employees in the morning. This system had cost a cool 400M USD, not including labor and upgrades for round 1. Round 2 was another 100M to add a storage buffer to each store because they'd failed to account for the fact that their internet connections at the store and the outbound pipe from the DC wasn't capable of running the public facing e-commerce and streaming all the video data to every store in realtime. Typical massive enterprise clusterfuck.

Then security gets involved. Each device at stores had a different address on a private megawan. The stores didn't generally phone home, home phoned them as an access control measure; stores calling the DC was verboten. This presented an obvious problem for the video system because it needed to pull updates.

The brilliant Infosys resources had a bright idea to solve this problem:

- Treat each device IP as an access key for that device (avg 15 per store per store).
- Verify the request ip, then issue a redirect with ANOTHER ip unique to that device that the firewall would ingress only to the video subnet
- Do it all with the F5

A few months later, the networking team comes back and announces that after months of work and 10s of people years they can't implement the solution because iRules have a size limit and they would need more than 60,000 lines or 15,000 rules to implement it. Sad trombones all around.

Then, a wild DBA appears, steps up to the plate and says he can solve the problem with the power of ORACLE! Few months later he comes back with some absolutely batshit solution that stored the individual octets of an IPV4, multiple nested queries to the same table to emulate subnet masking through some temp table spanning voodoo. Time to complete: 2-4 minutes per request. He too eventually gives up the fight, sort of, in that backhanded way DBAs tend to do everything. I wish I would have paid more attention to that abortion because the rationale and its mechanics were just staggeringly rube goldberg and should have been documented for posterity.

So I catch wind of this sitting in a CAB meeting. I hear them talking about how there's "no way to solve this problem, it's too complex, we're going to need a lot more databases to handle this." I tune in and gather all it really needs to do, since the ingress firewall is handling the origin IP checks, is convert the request IP to video ingress IP, 302 and call it a day.

While they're all grandstanding and pontificating, I fire up visual studio and:

- write a method that encodes the incoming request IP into a single uint32
- write an http module that keeps an in-memory dictionary of uint32,string for the request, response, converts the request ip and 302s the call with blackhole support
- convert all the mappings in the spreadsheet attached to the meetings into a csv, dump to disk
- write a wpf application to allow for easily managing the IP database in the short term
- deploy the solution one of our stage boxes
- add a TODO to eventually move this to a database

All this took about 5 minutes. I interrupt their conversation to ask them to retarget their test to the port I exposed on the stage box. Then watch them stare in stunned silence as the crow grows cold.

According to a friend who still works there, that code is still running in production on a single node to this day. And still running on the same static file database.

#TheValueOfEngineers

Every year my team have an award ceremony for stupid things we do throughout the year. There is only really one worth highlighting this year
Alzheimer’s award- given to one of our DBAs for this svn commit message: “updated the comment block but forgot to make the code change”

Biggest challenge I overcame as dev? One of many.

Avoiding a life sentence when the 'powers that be' targeted one of my libraries for the root cause of system performance issues and I didn't correct that accusation with a flame thrower.

What the accusation? What I named the library. Yep. The *name* was causing every single problem in the system.

Panorama (very, very expensive APM system at the time) identified my library in it's analysis, the calls to/from SQLServer was the bottleneck

We had one of Panorama's engineers on-site and he asked what (not the actual name) MyLibrary was and (I'll preface I did not know or involved in any of the so-called 'research') a crack team of developers+managers researched the system thoroughly and found MyLibrary was used in just about every project. I wrote the .Net 1.1 MyLibrary as a mini-ORM to simplify the execution of database code (stored procs, etc) and gracefully handle+log database exceptions (auto-logged details such as the target db, stored procedure name, parameter values, etc, everything you'd need to troubleshoot database errors). This was before Dapper and the other fancy tools used by kids these days.

By the time the news got to me, there was a team cobbled together who's only focus was to remove any/every trace of MyLibrary from the code base. Using Waterfall, they calculated it would take at least a year to remove+replace MyLibrary with the equivalent ADO.Net plumbing.

In a department wide meeting:

DeptMgr: "This day forward, no one is to use MyLibrary to access the database! It's slow, unprofessionally named, and the root cause of all the database issues."
Me: "What about MyLibrary is slow? It's excecuting standard the ADO.Net code. Only extra bit of code is the exception handling to capture the details when the exception is logged."
DeptMgr: "We've spent the last 6 weeks with the Panorama engineer and he's identified MyLibrary as the cause. Company has spent over $100,000 on this software and we have to make fact based decisions. Look at this slide ... "
<DeptMgr shows a histogram of the stacktrace, showing MyLibrary as the slowest>
Me: "You do realize that the execution time is the database call itself, not the code. In that example, the invoice call, it's the stored procedure that taking 5 seconds, not MyLibrary."
<at this point, DeptMgr is getting red-face mad>
AreaMgr: "Yes...yes...but if we stopped using MyLibrary, removing the unnecessary layers, will make the code run faster."
<typical headknodd-ers knod their heads in agreement>
Dev01: "The loading of MyLibrary takes CPU cycles away from code that supports our customers. Every CPU cycle counts."
<headknod-ding continues>
Me: "I'm really confused. Maybe I'm looking at the data wrong. On the slide where you highlighted all the bottlenecks, the histogram shows the latency is the database, I mean...it's right there, in red. Am I looking at it wrong?"
<this was meeting with 20+ other devs, mgrs, a VP, the Panorama engineer>
DeptMgr: "Yes you are! I know MyLibrary is your baby. You need to check your ego at the door and face the facts. Your MyLibrary is a failed experiment and needs to be exterminated from this system!"

Fast forward 9 months, maybe 50% of the projects updated, come across the documentation left from the Panorama. Even after the removal of MyLibrary, there was zero increases in performance. The engineer recommended DBAs start optimizing their indexes and other N+1 problems discovered. I decide to ask the developer who lead the re-write.

Me: "I see that removing MyLibrary did nothing to improve performance."
Dev: "Yes, DeptMgr was pissed. He was ready to throw the Panorama engineer out a window when he said the problems were in the database all along. Didn't you say that?"
Me: "Um, so is this re-write project dead?"
Dev: "No. Removing MyLibrary introduced all kinds of bugs. All the boilerplate ADO.Net code caused a lot of unhandled exceptions, then we had to go back and write exception handling code."
Me: "What a failure. What dipshit would think writing more code leads to less bugs?"
Dev: "I know, I know. We're so far behind schedule. We had to come up with something. I ended up writing a library to make replacing MyLibrary easier. I called it KnightRider. Like the TV show. Everyone is excited to speed up their code with KnightRider. Same method names, same exception handling. All we have to do is replace MyLibrary with KnightRider and we're done."
Me: "Won't the bottlenecks then point to KnightRider?"
Dev: "Meh, not my problem. Panorama meets primarily with the DBAs and the networking team now. I doubt we ever use Panorama to look at our C# code."

Needless to say, I was (still) pissed that they had used MyLibrary as dirty word and a scapegoat for months when they *knew* where the problems were. Pissed enough for a flamethrower? Maybe.

Worst collaboration experience story?

I was not directly involved, it was a Delphi -> C# conversion of our customer returns application.

The dev manager was out to prove waterfall was the only development methodology that could make convert the monolith app to a lean, multi-tier, enterprise-worthy application.

Starting out with a team of 7 (3 devs, 2 dbas, team mgr, and the dev department mgr), they spent around 3 months designing, meetings, and more meetings. Armed with 50+ page specification Word document (not counting the countless Visio workflow diagrams and Microsoft Project timeline/ghantt charts), the team was ready to start coding.
The database design, workflow, and UI design (using Visio), was well done/thought out, but problems started on day one.
- Team mgr and Dev mgr split up the 3 devs, 1 dev wrote the database access library tier, 1 wrote the service tier, the other dev wrote the UI (I'll add this was the dev's first experience with WPF).
- Per the specification, all the layers wouldn't be integrated until all of them met the standards (unit tested, free from errors from VS's code analyzer, etc)
- By the time the devs where ready to code, the DBAs were already tasked with other projects, so the Returns app was prioritized to "when we get around to it"

Fast forward 6 months later, all the devs were 'done' coding, having very little/no communication with one another, then the integration. The service and database layers assumed different design patterns and different database relationships and the UI layer required functionality neither layers anticipated (ex. multi-users and the service maintaining some sort of state between them).

Those issues took about a month to work out, then the app began beta testing with real end users. App didn't make it 10 minutes before users gave up. Numerous UI logic errors, runtime errors, and overall app stability. Because the UI was so bad, the dev mgr brought in one of the web developers (she was pretty good at UI design). You might guess how useful someone is being dropped in on complex project , months after-the-fact and being told "Fix it!".

Couple of months of UI re-design and many other changes, the app was ready for beta testing.

In the mean time, the company hired a new customer service manager. When he saw the application, he rejected the app because he re-designed the entire returns process to be more efficient. The application UI was written to the exact step-by-step old returns process with little/no deviation.

With a tremendous amount of push-back (TL;DR), the dev mgr promised to change the app, but only after it was deployed into production (using "we can fix it later" excuse).

Still plagued with numerous bugs, the app was finally deployed. In attempts to save face, there was a company-wide party to celebrate the 'death' of the "old Delphi returns app" and the birth of the new. Cake, drinks, certificates of achievements for the devs, etc.

By the end of the project, the devs hated each other. Finger pointing, petty squabbles, out-right "FU!"s across the cube walls, etc. All the team members were re-assigned to other teams to separate them, leaving a single new hire to fix all the issues.

Tonight do not forget to raise a glass to Ops, oncall developers, DBAs and devops, field engineers, warehouse operators. Thanks to them, we can afford to meet the new year properly.

They are working tonight so we wouldn't have to.

Cheers

Client: *uses Oracle on AWS RDS*

Ora: *licence about to expire*

Client: we need a cheaper and equally performant solution.

AWS: we have an outstanding Aurora db! Its performance is flawless!

Client: shut_up_and_take_my_money.png

Engineers: *migrating to Aurora mysql*
Engineers: fine-tuning the db for 2 months, adapting app code.
Aurora: *shits its pants during test ramp-up, delivers ~70% of Ora perf*

==========

Client: we need to do better!

AWS: try Aurora Postgresql!

Engineers: *migrating to Aurora postgresql*
Engineers: fine-tuning the db for 1 month.
Aurora: *shits its pants during test ramp-up, delivers ~40% of Ora perf*

AWS: let me see...
AWS DBAs: *fine-tuning the db for another 2 months*
Aurora: *survives ramp-up, delivers ~70% Ora perf*

AWS DBAs: your application is wrong.
We: Ora didn't complain about that...

AWS DBAs: umm.. Err.. Then your load test is wrong
We: ora wasn't complaining about that either...

AWS DBAs: errr.. Ummm.. Oh, I've got it! Your queries aren't optimized!
We: we cannot change queried - they are hardcoded by our framework's vendor

AWS DBAs: well there you go! Your queries aren't optimized! It's your fault, not aurora's

yyeeahhhh... Riight...

From my xp, aws support and aws engineers ALWAYS try to put the blame on a client. Always. Even when they're obviously wrong.

Today was a manic-depressive kind of day. Spent the morning helping some developers with getting their code to run a stored procedure to drop old partitions, but it wasn't working on their end. It was a fairly simple proc. But working with partitions is a little like working with an array. I figured out that they were passing the wrong timestamp, and needed to add +1 to delete the right partition. Got that sorted out, and things were good. Lunch time.

After lunch I did some busy work, and then the PO comes up at about 2PM and says he's assigned some requests to me. The first was just attaching some scripts. Easy. The second, the user wants a couple of schemas exported ... at 6PM. I've been in the office since 6:45AM.

While I'm setting up some commands to run for the data export, a BA walks up and asks if I'm filling in for another DBA who is out for a few weeks. Yep. There's a change request that hasn't been assigned, and he normally does the work. I ask when it's due. Well, the pre-implementation was supposed to be done in the morning, but it wasn't, and we're in the implementation window ... half way through. I bring up the change task, and look at. Create new schema and users. That's all it says. The BA laughs. I tell I need more to go on. 10 minutes later he sends an email with the information. There's only two hours left in the window, and I can only use half of it, because the production guys have to their stuff, and we're in their window. Now I'm irritated, because I'm new to Oracle, and it's an unforgiving mistress. Fortunately, another DBA says he'll do it, so that we can get it done in time. But can't work it either, because Dev DBAs don't have access to QA, and the process required access for this task. Gets shelved until the access issue is resolved. It's now after 4:15PM. I'm going to in traffic with that 6PM deadline.

I manage to get home and to the computer by 5:45PM. Log in. Start VPN. Box pops on screen. Java needs to update. I chose skip update. Box pops up again. It won't let me log in until Java is current. Passed.

I finally get logged in, and it's 6:10PM. I'm late getting the job started. I pull up Putty and log into the first box, and paste my pre-prepared command in the command line and hit error. Command not found. I'm tired, so it's a moment to sink in. I don't have time for this.

I log into DBArtisan and pull up the first data base, use the wizard to set the job, and off it goes. Yay. Bring up the second database, and have enter the connect info. Host not found. Wut? Examine host name. Yep, it's correct. Try a different method. Host not found. Go back to Putty. Log in. Past string. Launch. Command not found. Now my brain is quitting on me. Why now? It's after 6:30PM. Fiddle with some settings, reset $Oracle home. Try again. Yay. It works. I'm done. It's after 7PM.

There is nothing like technology to snatch the euphoria of a success away from you. It's a love-hate thing, but I wouldn't trade it for anything else. I'm done. Good night.

Still dealing with the web department and their finger pointing after several thousand errors logged.

SeniorWebDev: “Looks like there were 250 database timeout errors at 11:02AM. DBAs might want to take a look.”

I look at the actual exceptions being logged (bulk of the over 1,600 logged errors)..

“Object reference not set to an instance of an object.”

Then I looked the email timestamp…11:00AM. We received the email notification *before* the database timeout errors occurred.

I gather some facts…when the exceptions started, when they ended, and used the stack trace to find the code not checking for null (maybe 10 minutes of junior dev detective work). Send the data to the ‘powers that be’ and carried on with my daily tasks.
I attached what I found (not the actual code, it was changed to protect the innocent)
Couple of hours later another WebDev replied…

WebDev: “These errors look like a database connectivity issue between the web site and the saleitem data service. Appears the logging framework doesn’t allow us to log any information about the database connection.”

FRACK!!...that Fracking lying piece of frack! Our team is responsible for the logging framework. I was typing up my response (having to calm down) then about a minute later the head DBA replies …

DBA: “Do you have any evidence of this? Our logs show no connectivity issues. The logging framework does have the ability to log an extensive amount of data regarding the database transaction. Database name, server, login, command text, and parameter values. Everything we need to troubleshoot. This is the link to the documentation …. If you implement the one line of code to gather the data, it will go a long way in helping us debug performance and connectivity issue. Thank you.”

DBA sends me a skype message “You’re welcome :)”

Ahh..nice to see someone else fed up with their lying bull...stuff.

Had an internet/network outage and the web site started logging thousands of errors and I see they purposely created a custom exception class just to avoid/get around our standard logging+data gathering (on SqlExceptions, we gather+log all the necessary details to Splunk so our DBAs can troubleshoot the problem).

If we didn't already know what the problem was, WTF would anyone do with 'There was a SQL exception, Query'? OK, what was the exception? A timeout? A syntax error? Value out of range? What was the target server? Which database? Our web developers live in a different world. I don't understand em.

Ever had a day that felt like you're shoveling snow from the driveway? In a blizzard? With thunderstorms & falling unicorns? Like you shovel away one m² & turn around and no footprints visible anymore? And snow built up to your neck?

Today my work day was like that.. xcept shit..shit instead of pretty & puffy snow!!

Working on things a & b, trying to not mess either one up, then comes shit x, coworker was updating production.. ofc something went wrong.. again not testing after the update..then me 'to da rescue'.. :/ hardly patch things up, so it works..in a way.. feature c still missing due to needed workarounds.. going back to a and b.. got disrupted by the same coworker who is nver listening, but always asking too much..

And when I think I finally have the b thing figured out a f-ing blocker from one of our biggest clients.. The whole system is unresponsive.. Needles to say, same guy in support for two companies (their end), so they filed the jira blocker with the wrong customer that doesn't have a SLA so no urgent emails..and then the phone calls.. and then the hell broke loose.. checking what is happening.. After frantic calls from our dba to anyone who even knows that our customer exists if they were doing sth on the db.. noup, not a single one was fucking with the prod db.. The hell! Materialised view created 10 mins ago that blocked everything..set to recreate every 10 minutes..with a query that I am guessing couldn't even select all that data in under 15.. dafaaaq?! Then we kill it..and again it is there.. We found out that customers dbas were testing something on live environment, oblivious that they mamaged to block the entire db..
FML, I'm going pokemon hunting.. :/ codename for ingress n beer..

Management directed a 2-month project including 5 developers, 3 DBAs, plus Q&A to replace a SOAP service that retrieved data from a single table. End result, project lasted 9+ months, 5 spectacular failed 3:00AM deployments, and a WCF service that retrieved data from a single table. Justification? Management wanted to eliminate SOAP, because SOAP uses XML and XML is slow. Thank goodness no one opened up Fiddler to see how WCF communicates.

A dev found a bug I created where I set a SQL parameter name to @OrderID instead of the expected @Order. The standard is @OrderID, there is one stored proc where it's @Order.
Oops...I didn't catch it because the integration test didn't cover that area of the code. My mistake...I should have checked...I take complete responsibility for the screw up.

He let me know by email..
"When refactoring, from now on check the stored procedure parameters, there are a few that don't follow the standard."

I was like "from now on..."? ...wow....bold comment from someone responsible for code that doesn't check for nulls, doesn't log errors, and relies on exceptions for flow control. You wouldn't even have known about the error if I didn't modify your code to log the error (the try..except returned false)

I really wanted to reply ...

"Fixed. From now on, when you come across those easily found bugs, go head and fix it, write a test, and move on. Don't send a condescending email to me, my boss, your boss, all the DBAs, and the entire fracking order processing team. Thanks."

But..I thanked him for finding and letting me know...we're a team..blah blah blah..

Frack..people suck.

!coding

I used to be a sysadmin, which meant I was in charge of quarterly server patching. My team managed about 2500 servers, running various flavors of linux and legacy unix. The vast majority(95% or more) ran Linux(SLES). Our maintenance window was always in the overnight-- 10pm to 6am --so the stroke of 10pm would be a massive cascade of patching commands sent to hundreds of servers.

Before I was brought into the process, it made use of the automation product we were tasked by mgmt to use: Bigfix. It's a real piece of shit. Though we had 2500 or so servers, this environment was dominated by windows. All our vcenter servers ran it, and more importantly, our bigfix nodes were all windows machines. That meant that while we're trying to patch, the bigfix servers would get patched by the windows team. This would cause lots of failed and timed out patching, because the windows admins never quite understood that taking down the automation infrastructure would cause problems.

As such, I got tired of depending on a bunch of button-pushing checkbox-clickers who didn't know shit about shit, so I started writing an ssh-wrapped patching system. By the time I left for my current job, patching had been reduced to a single command to initiate each group's patching and reboots, and an easy check to see when servers come back up. So usually, the way it worked out was that I would send patching orders to 750 machines or so, and within about 5 minutes, they would all be done patching, and within another 20 minutes all the ones that required rebooting but about 5 would be done rebooting.

The "all-nighter" which happened every time was waiting for oracle servers to run timed fscks against a dozen or so large filesystems per server, because they were all on ext3/4, which eats complete shit. Then, several hours later, as they finished, I would have to call the DBAs to tell them to validate their shitty servers.

Worst architecture I've seen?

The worst (working here) follow the academic pattern of trying to be perfect when the only measure of 'perfect' should be the user saying "Thank you" or one that no one knows about (the 'it just works' architectural pattern).

A senior developer with a masters degree in software engineering developed a class/object architecture for representing an Invoice in our system. Took almost 3 months to come up with ..

- Contained over 50 interfaces (IInvoice, IOrder, IProduct, etc. mostly just data bags)
- Abstract classes that implemented the interfaces
- Concrete classes that injected behavior via the abstract classes (constructors, Copy methods, converter functions, etc)
- Various data access (SQL server/WCF services) factories

During code reviews I kept saying this design was too complex and too brittle for the changes everyone knew were coming. The web team that would ultimately be using the framework had, at best, vague requirements. Because he had a masters degree, he knew best.

He was proud of nearly perfect academic design (almost 100% test code coverage, very nice class diagrams, lines and boxes, auto-generated documentation, etc), until the DBAs changed table relationships (1:1 turned into 1:M and M:M), field names, etc, and users changed business requirements (ex. concept of an invoice fee changed the total amount due calculation, which broke nearly everything).

That change caused a ripple affect that resulted in a major delay in the web site feature release.

By the time the developer fixed all the issues, the web team wrote their framework and hit the database directly (Dapper+simple DTOs) and his library was never used.

I absolutely hate Database First. Sorry, DBAs.

We are researching enhancing our current alerting system (we use Splunk) to be 'smarter' about who is emailed/texted/whatever when there are problems in our applications.

Currently, if there are over 50 errors logged within a 15 minute period, a email/phone/text blast to nearly 100 individuals ranging from developers, network admins, DBAs, and vice presidents.

Our plan is to group errors by team and let each team manage their own applications. Alert on 1 error, 5, 500...we don't care, let the team work out the particulars.

The trick was interfacing with Splunk's API (that's a long rant by itself)

In about a day or so I was able to use Splunk's WebHook feature to notify a WebAPI service I threw together to send myself an email with details about the underlying data (simulating the kind of alert we would send to the team)

I thought ...cool... it worked. Show it off to the team, most thought it was a good start, except one:

Dev: "The errors are not grouped by team."
Me: "No, I threw the webapi service together to demonstrate how we can extract the splunk bits to get access to the teams"
Dev: "Well...this won't work at all."
Me: "Um..what?"
Dev: "The specification c l e a r l y states the email will be team based. This email was only sent to you and has all the teams and their applications"
Me: "Um...uh...the service can, if we want to go using a service route. Grouping by team name is easy using a LINQ query. I just through this service together yesterday."
Dev: "I don't know. Sounds like I need to schedule a meeting to discuss what you are proposing. I don't think emailing all that to everyone is a good idea."

WTF! Did you not listen to what I said?!!!

Oh well..the dev's proposal is to use splunk's email notification and custom Exchange rules with callbacks into splunk that resend...oh good lord ...a fracking rube goldberg of a config nightmare ...

I suspect we'll go the service route once I finish the service before the meeting.

So.. Yeah.. A huge corp decided they no longer need a dba. For at least 2 months, starting DEC.

Ora, enterprise features - all the way in. Meaning only dbas have the knowledge

I've got some corn to pop. Anyone fancy to join?

On this episode of “My Work Life” we will once again find that the code is fine, the DBAs just did something brainless again.

Product owner: when will you have that script written and submitted to DBAs?
Me: if nobody bugs me, hopefully COB today.
Product owner: great!
*user support*
*user support*
*user support*
*user support*
*user support*
Team lead: write this other script.
Me: I just finished user support, I have another script to write first. Does yours take priority?
Team lead: yes.
Me: ok....*sigh*
*writes 2nd script*
*submits 2nd script to DBAs*
Product Owner: you done?
Me: *face palm* ......NO!

Spent a couple of hours in the range with the team today.

I wish we invited DBAs too. Still targets get boring after the first couple dozen rounds

Story of my first successful project

Being part of a great team, I've shared in a lot of successes, one I am particularly proud of is my first attempt to use agile methodologies in a deeply waterfall-managment culture.

Time was June/July-ish and we applied for a national quality award where one key element in the application stated how well we handled customer complaint resolution.
While somewhat true (our customer service is the top-shelf good stuff), we did not have a systematic process in resolving customer complaints. Long story short,
the VP lied on her section of the application. Then came the 'emergency', borderline panic meeting (several VPs, managers, etc) to develop a process to better manage
complaints before the in-house inspection in December.
As most top priority projects go, the dev manager allocated 3 developers, 2 DBAs, and any/all network admins we would need (plus all the bureaucratic management that wanted their thumb in the pie).
Fast forward to August, after many, many planning meetings, lost interest, new shiny bouncing balls, I was the only one left on the project. The VP runs into the dev manager in the hallway and asks "Is my program done yet? If its not ready before December with report-able data, we will not win the award."
The <bleep> hit the fan...dev manager comes by...
Frank: "How the application coming along? Almost done?"
Me:"No, haven't really started coding. You moved Jake and Tom over to James's team, Tina quit, and you've had me sidetracked helping other teams because the DBAs are too busy."
Frank: "So, it's excuses. You really think the national quality award auditors care about your excuses? The specification design document has been done for months. This is unacceptable."
Me: "The VP finished up her section yesterday and according to the process, we can't start coding until the document is signed off."
Frank: "Holy f<bleep>ing sh<bleep>t! No one told you *you* couldn't start. You know how to create tables and write code."
Me: "There is no specification to write to. The design document is all about how they plan on reporting the data, not how call agents will be using the application to serve customers."
Frank: "The f<bleep> it isn't. F<bleep>ing monkeys could code against that specification, I helped write it! NO MORE F<bleep>ING EXCUSES! This is your top priority from now on!"

I was 'cleared' to work directly with the call center manager and the VP to develop a fully integrated customer complaint management system before December (by-passing any of the waterfall processes that would get in the way).

I had heard about this 'agile' stuff, attended a few conference tracks on the subject, read the manifesto, and thought "I could do this.".

Over the next month, I had my own 'sprints' and 'scrums' with the manager (at the time, 'agile' was a dirty word so I had to be careful of my words and what info I shared) and by the 2nd iteration had a working prototype.
Feature here, feature there (documenting the 'whys' and 'whats' along the way), and by October, had a full deployed application.

Not thinking I would get a parade or anything, the dev manager came back from a meeting where the VP was showing off the new app to the other VPs (and how she didn't really 'lie' on the application)

Frank: "Everyone is pleased how well the project turned out, except one thing. Erin said you bothered him too much with too many questions."
Me: "Bothered? Did he really say that?"
Frank: "No, not directly, but he said you would stop by his office every day to show him your progress and if he needed you to change anything. You shouldn't have done that."
Me: "Erin really seemed to like the continuous feedback. What we have now is very different than what we started with."
Frank: "Yes, probably because you kept bothering him and not following the specification document. That is why we spend so much time up front in design is so we don't waste management's time, which is exactly what you did."
Me: "We beat the deadline by two months, so I don't think I wasted anyone's time. In fact, this is kind of a big win for us, right?"
Frank: "Not really. There was breakdown in the process. We need better focus on the process, not in these one-hit-wonders."

End the end, the company won the award (mgmt team got to meet the vice president, yes the #2 guy). I know I played a very small, somewhat insignificant role in that victory, I was extremely proud to be part of the team.

Before my vacation I’d been chatting with one of our dbas about an etl tool we needed for a customer we’d already signed all the contracts with saying we would provide one for a historical database of old data. They had been looking at one from SAP but in typical fashion a license was worth more than the actual contract.

Anyway long story short on the weekend before I went back to work I rattled together a little python proof of concept using a couple of azure databases and when I went back demo’d it to the pm and dba they loved it and we built on the poc to have a working loader which saved us about £30k by not buying the SAP product and just wrote our own.

This friend of mine accidentally deleted client data in their production database, DBAs gotta work paid overtime to fix the mess...at least he only got away with a written warning. Dude coulda been fired on the spot

DBAs have been receiving an alert every 5 minutes for the last two hours regarding a blocking SPID in production. I still have to tell them about the alert and ask them to resolve the issue...

- Every specialist is looking after his area of expertise

- Everyone is a specialist of everything and shall work on everything

- Every specialist is looking after his area of expertise, making improvements and automations in his area

- Everyone is a specialist of everything and is looking after everything, automating everything (devops)

- Everyone is a specialist of everything and is looking after everything, automating everything, in all the environments (SRE)

- ... I wonder what's next...

I miss the good old days when developers could be developers and rely on DBAs, sysadmins and networkists to do their job well. I miss the days when developers were developing applications, sytems, modules,.. Not troubleshooting ELBs, RDS latencies or building monitoring for servers.

It be cool our if SVN got fixed so I could commit changes again. Or while it's down look at other vc options, maybe git?

Nudge, nudge wink, wink.

But in all seriousness wtf did the dba's do to break SVN? Logging my changes in a spreadsheet sucks raw balls.

I'm working on broadcasting changes in a SQL Server db using web sockets, but trying to not install anything because then I'd have to get our DBAs involved...

Spent hours trying to package a little node app that broadcasts the changes as an exe using nexe, realized for the most part it just compiles node from source, and the outputted binary didn't end up running, anyway.

Then it hit me; I can just run the node exe without installing it. Now I just have to get this service broker to work...

Just had the best meeting since I started here. The DBAs couldn't attend so we decided to postpone the meeting for two weeks. After that we started talking about beer and now I have 5 names of different beers that I want to try :-)