Yesterday the web site started logging an exception “A task was canceled” when making a http call using the .Net HTTPClient class (site calling a REST service).

Emails back n’ forth ..blaming the database…blaming the network..then a senior web developer blamed the logging (the system I’m responsible for).

Under the hood, the logger is sending the exception data to another REST service (which sends emails, generates reports etc.) which I had to quickly re-direct the discussion because if we’re seeing the exception email, the logging didn’t cause the exception, it’s just reporting it. Felt a little sad having to explain it to other IT professionals, but everyone seemed to agree and focused on the server resources.

Last night I get a call about the exceptions occurring again in much larger numbers (from 100 to over 5,000 within a few minutes). I log in, add myself to the large skype group chat going on just to catch the same senior web developer say …
“Here is the APM data that shows logging is causing the http tasks to get canceled.”

FRACK!

Me: “No, that data just shows the logging http traffic of the exception. The exception is occurring before any logging is executed. The task is either being canceled due to a network time out or IIS is running out of threads. The web site is failing to execute the http call to the REST service.”

Several other devs, DBAs, and network admins agree.

The errors only lasted a couple of minutes (exactly 2 minutes, which seemed odd), so everyone agrees to dig into the data further in the morning.

This morning I login to my computer to discover the error(s) occurred again at 6:20AM and an email from the senior web developer saying we (my mgr, her mgr, network admins, DBAs, etc) need to discuss changes to the logging system to prevent this problem from negatively affecting the customer experience...blah blah blah.

FRACKing female dog!

Good news is we never had the meeting. When the senior web dev manager came in, he cancelled the meeting.

Turned out to be a hiccup in a domain controller causing the servers to lose their connection to each other for 2 minutes (1-minute timeout, 1 minute to fully re-sync). The exact two-minute burst of errors explained (and proven via wireshark).

People and their petty office politics piss me off.

The bossman asked if our signup service sends an automated email after we successfully process someone's payment or when we promote them to full customer.

That sounds like a simple query, yeah?

Well.

Here's some background:

We have four applications; one in React, three in Rails. I'll replace their names to retain some anonymity.
1) "IceSkate" is the React app, and it's a glorified signup form. (I wrote this one.)
2) "Bogan" is the main application, and is API-only; its frontend has been long since deprecated by the following two:
3) "Bum" is a fork of "Bogan" that has long since diverged. It now contains admin-only tools.
4) "Kulkuri" is also a fork of "Bogan" that has long since diverged. It now contains tools specifically for customers, which they can access.

All but IceSkate (obv) share a database.

Here's how signups happen:

Signups come in from IceSkate, which hits a backend API on Bogan. Bogan writes the data to the database, charges the card immediately, and leaves the signup for moderation.

And here's how promotion from signup to customer happens:

Bum has a view allowing admins to validate, modify, and "promote" a signup to a full customer. Upon successful promotion, Bum calls "ServerWrap", a module which calls actions on the other applications; in this case: Bogan.

Bogan routes execution through three separate models before calling "ServerWrap" again, this time calling KulKuri.

Finally, KulKuri actually creates the customer!

After KulKuri finishes creating the customer, execution resumes on Bogan, which then returns, causing execution to resume on Bum. Bum then runs through several other models, references the newly-created customer object (as all three share a database), and ... updates the customer with its current data, and then updates the signup object. After all of this, it finally shows the admin the "new customer" view.

It took me 25 minutes to follow the chain of calls, and I still don't know quite what's going on. I have no idea if any of it sends an email or not -- I didn't see any signs of this, but I very easily could have overlooked something.

So, to answer bossman's question... I asked the accounting people if they send the email manually. If they don't, it's automatic, which means I missed something and get to burrow through that mess all over again!

I really hope I missed something; otherwise I need to figure out how and where (and when!) to send the email...

just...
errrrgghh

Database admins out there?

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

Been working on a new project for the last couple of weeks. New client with a big name, probably lots of money for the company I work for, plus a nice bonus for myself.

But our technical referent....... Goddammit. PhD in computer science, and he probably. approved our project outline. 3 days in development, the basic features of the applications are there for him to see (yay. Agile.), and guess what? We need to change the user roles hierarchy we had agreed on. Oh, and that shouldn't be treated as extra development, it's obviously a bug! Also, these features he never talked about and never have been in the project? That's also a bug! That thing I couldn't start working on before yesterday because I was still waiting the specs from him? It should've been ready a week ago, it's a bug that it's not there! Also, he notes how he could've developes it within 40 minutes and offered to sens us the code to implement directly in our application, or he may even do so himself.... Ah, I forgot to say, he has no idea on what language we are developing the app. He said he didn't care many times so far.

But the best part? Yesterday he signales an outstanding bug: some data has been changed without anyone interacting. It was a bug! And it was costing them moneeeeey (on a dev server)! Ok, let's dig in, it may really be a bug this time, I did update the code and... Wait, what? Someone actually did update a new file? ...Oh my Anubis. HE did replace the file a few minutes before and tried to make it look like a bug! ..May as well double check. So, 15 minutes later I answer to his e-mail, saying that 4 files have been compromised by a user account with admin privileges (not mentioning I knee it was him)... And 3 minutes later he answered me. It was a message full of anger, saying (oh Lord) it was a bug! If a user can upload a new file, it's the application's fault for not blocking him (except, users ARE supposed to upload files, and admins have been requestes to be able to circumvent any kind of restriction)! Then he added how lucky I was, becausw "the issue resolved itself and the data was back, and we shouldn't waste any more yime.on thos". Let's check the logs again.... It'a true! HE UPLOADED THE ORIGINAL FILES BACK! He... He has no idea that logs do exist? A fucking PhD in computer science? He still believes no one knows it was him....... But... Why did he do that? It couldn't have been a mistake. Was he trying to troll me? Or... Or is he really that dense?

I was laughing my ass of there. But there's more! He actually phones my boss (who knew what had happened) to insult me! And to threaten not dwell on that issue anymore because "it's making them lose money". We were both speechless....

There's no way he's a PhD. Yet it's a legit piece of paper the one he has. Funny thing is, he actually manages to launch a couple of sort-of-nationally-popular webservices, and takes every opportunity to remember us how he built them from scratch and so he know what he's saying... But digging through google, you can easily find how he actually outsurced the development to Chinese companies while he "watched over their work" until he bought the code

Wait... Big ego, a decent amount of money... I'm starting to guess how he got his PhD. I also get why he's a "freelance consultant" and none of the place he worked for ever hired him again (couldn't even cover his own tracks)....

But I can't get his definition of "bug".
If it doesn't work as intended, it's a bug (ok)
If something he never communicated is not implemented, it's a bug (what.)
If development has been slowed because he failed to provide specs, it's a bug (uh?)
If he changes his own mind and wants to change a process, it's a bug it doesn't already work that way (ffs.)
If he doesn't understand or like something, it's a bug (i hopw he dies by sonic diarrhoea)

I'm just glad my boss isn't falling for him... If anything, we have enough info to accuse him of sabotage and delaying my work....

Ah, right. He also didn't get how to publish our application we needes access to the server he wantes us to deploy it on. Also, he doesn't understand why we have acces to the app's database and admin users created on the webapp don't. These are bugs (seriously his own words). Outstanding ones.

Just..... Ffs.

Also, sorry for the typos.

Hello not a rant,

Are there MS SQL Server admins here who self taught and learn thru self study?

I work in a company where they use MS SQL Server as the database. I would love to 'understand' how to write efficient queries, and how things really work, not just selecting and joining table blindly and not understanding how things work.

Would you recommend how you understand MS SQL Server, or what learning path you took?

Thank you. I would appreciate any suggestions and comments.

Always check if you are on the test or the prod system at least 1000 times before cleaning up the customers database.

This happend twice to my team.... At least our customers admins were pretty chill and said they learned something from this.

Had trouble to connect to our MySQL database, so I decided to open a ticket to the Database admins. At least they are pros and I'm sure they'll help me:

"Hey guys, I have trouble connecting to [Hostname]. I guess it's a firewalling issue would you take a look? Attached are screenshots, saying hostname not found.

Answer:
Hey Dominique, are you sure the password you used is correct? Is it yours or the sysuser pw what you sent to the server? How did you send it?

Me: (kind of confused) Hey dear admin, did you look at my error message? It says Hostname not found. What do you think how I provided any credentials?

Support: yes, I saw your screenshot and don't see any password entry. That's why I asked!

Me: Well, than... ok... go and search for another job. Yeah and consider fucking yourself. Kisses.

It’s fascinating how IT and database admins want us to drop tables that we don’t use anymore... with read-only access!

3 database-admins are walking into a bar. Shortly the leave without ordering or even sit down for one sinple reason: they couldn't find a table.

Hey, can anyone give me an idea on how syncing works?
(Been searching for past few hours, didn't went past some weird advance server admins stuff or cloud backup ads.)

All i wish to do is to make a service for my app that will run every night and upload/delete /modify (basically , sync) my app's local database to the server.

Just gimme a rough idea on the algorithm to use, i have done till running the service every night( but what exactly it should do, i have no idea)

Sometimes I have to connect to production database and alter my dev environment so I can “log in” as a user and see what’s wrong with their account. Once in a while there is a legitimate website issue that is unique to that user’s profile. Other times it’s user error, like the user not understanding that they have to connect their membership to their online account (they think signing up for an account will connect it automatically).

I don’t like circumventing the user’s log in like this, but sometimes it’s necessary since the website is so confusing. I inherited this website, so many of the problems were formed way before I took over.

My stakeholders want a log in as user feature for website admins to use. My manager and PM don’t think that’s a good idea right now since there are over two dozen people with admin access and admin access means access to everything in the admin (there aren’t options to give permissions as needed).