Stakeholder: Users are unable to buy tickets on the website. IT says Azure’s health check is showing an unhealthy status.

[It’s Sunday. Web Engineering is not on call so no one sees this right away.]

Stakeholder: IT restarted the Azure website twice, but users still can’t place orders.

Me: There was never an issue with the Azure site. That health check is inaccurate. There is a rewrite rule that sends the Azure supplied domain to our custom domain. The Azure health check doesn’t like that so it returns an unhealthy status. The problem is the ticketing server that the website has to communicate with. The ticketing server is overwhelmed and can’t handle more requests. IT should have checked the ticketing server’s logs. This has happened before and it’s never been an Azure issue. It’s a ticketing server issue.

Stakeholder and IT: Oops 😅

—-

JFC. Stop trying to make this web engineering’s problem. Stop trying to make it look like engineering dropped the ball. The ticketing server has experienced this issue multiple times. The ticketing server is maintained by a different team. The website’s symptoms are always the same and there are steps you need to take before you make the decision to restart the website, which will cause the website to show a blue screen of death that says 503 service unavailable for a few minutes. And we have a switch to shut off all transactions. Why do you not want to use it when it’s clear the website can’t process transactions???

I’m LOLing at the audacity of one of our vendors.

We contract with a vendor to build and maintain a website. Our network security team noticed there was a security breach of the vendor’s website. Our team saw that malicious users gained access to our Google Search console by completing a challenge that was issued to the vendor’s site.

At first, the vendor tried to convince us that their site wasn’t comprised and it was the Google search Console that was compromised. Nah dude. Our Search Console got compromised via the website you maintain for us. Luckily our network team was able to remove the malicious users from our search console.

That vendor site accepts credit card payments and displays the user’s contact info like address, email, and phone. The vendor uses keys that are tied to our payment gateway. So now my employer is demanding a full incident report from the vendor because their dropping the ball could have compromised our users’ data and we might be responsible for PCI issues.

And the vendor tried to shit on us even more. The vendor also generates vanity urls for our users. My employer decided to temporarily redirect users to our main site (non vendor) because users already received those links and in order to not lose revenue. The vendor’s solution is to build a service that will redirect their vanity urls to our main site. And they wanted to charge us $5000 usd for this. We already pay them $1000 a month already.

WTAF we are not stupid. Our network service team said we could make the argument that they do this without extra charge because it falls in the scope of our contract with them. Our network team also said that we could terminate the contract because the security breach means they didn’t render the service they were contracted to do. Guess it’s time for us to get our lawyer’s take on this.

So now it looks like my stakeholders want me to rebuild all of this in house. I already have a lot on my plate, but I’m going to be open to their requests because we are still in the debrief phase.

Other staff: I’m having trouble logging in to website A. My password doesn’t work.

[Me thinking: That’s weird. When I set up your account, the password worked. I told you to change it. So maybe you forgot your new password. We haven’t changed anything to about the login process.]

Me: I reset your password. [sends new password]

Other Staff: The new password doesn’t work. But I can log in with Google.

Me: 😶 Website A does not have sign in with Google. What website are you actually on???

We have a CMS that’s supposed to be simple to use so non-technical staff can make some webpages. A lot of it doesn’t require much brain work. Just duplicate a page and swap out text and images. But they keep forgetting how to log in to the website even though I shared written instructions on how to log in.

Recently, I told the Head of Engineering Manager that we should retire the CMS because it’s not intuitive to use and it doesn’t get used a lot. There used to be one dude who did it, but he left. So employee turnover plus no one using it a lot means folks don’t learn or forget. And they end up coming to Engineering for help with swapping out text and images.

Someone didn’t properly set the httpcookies domain for our staging and production websites. Yep, this was a C#/.NET site. The cookie domain for the staging site was set to the production domain instead of the staging domain (which was a subdomain). So if someone logged into the staging admin, that would also grant them access to production admin if they also had an account in the production site.

The staging site technically had an additional login to enter the site, but the username and password weren’t too hard to guess. It was like that for years until I was hired to be an in-house dev (the role was previously outsourced to a software development company).

The admin side of the website wasn’t very sophisticated. But there was enough personal identifying info for a hacker to do something with.

I don’t know how they weren’t hacked yet. Honestly, I’d tell my employer to go back to that software agency and ask for a refund and cite the shotty work.

Former classmate: Our alma mater is looking for alumni to participate in career day. Share what skills you need and the steps you took for your career path!

Me: Thanks for the invite. But I’m not a good role model for this.

FC: Why not? You’re a successful engineer!

Me: So I used my full tuition university scholarship on an art degree because I was too depressed after a long physical illness. Oh, and for some reason a lot of y’all assumed I went to a private uni when I went to the public uni. Then I went to graduate school immediately after and during a recession and ended up with tens of thousands in student debt. Then I did a lot of part time jobs before going to a shady coding bootcamp. I’m lucky to have encountered an advocate and a company willing to take me on as a junior dev. I’m pretty sure I was a diversity hire and I was definitely underpaid. I’m lucky to have moved on from there and to be thriving now. I’d tell the students to skip college (like I had considered) and go into a trade. And I’d also tell them a lot of life is luck and not just hard work.

FC: 😧

I wish Slack had a block feature. On a social Slack, someone lashed out me and started accusing me of horrible things. Admins did nothing after I complained and said my anger wasn’t in proportion to the situation. Fuck that.

The lasher out accused me of ableism, povertyism, and condoning human rights violations. It was so outrageous that even a bystander tried to intervene because lasher out was clearly acting out on a trigger and I had not done anything to deserve it.
I had this problem with the lasher out before, but this time they went too far.

So I have one less social platform to engage with. Good riddance. I’m not participating in a place that is not a safe space.

I thought Facebook would be the one I unplugged first.

I’m thankful for my team. I had an upsetting personal incident and it was affecting my ability to be present. I even missed a meeting. They are giving me the space I need and rescheduling other meetings for me. It’s hard for me to accept compassion because I haven’t experienced a lot of that when I needed it. I was very distressed at the beginning of the week, but now I’m starting to feel better and getting back to functioning.

Stakeholder: Is it possible for you to set up the website to automatically resubmit failed online orders? Last time there were failed orders, we tried submitting manually but a lot failed because they were tickets for the previous day.

Product Manager: What are your thoughts, Developer?

Me: This wouldn’t be worth the labor. It’s something that would rarely be used. There are very few orders that fail. I’d be surprised if it was even once a week. The recent bunch of order failures that SH is talking about happened because the ticketing server (separate from the website) couldn’t handle all the requests. Let’s say you had resubmission logic to try 3x before allowing the fail. It wouldn’t work because the server was overwhelmed already. Let’s say you had a background task to check for failures every ten minutes and resubmit those. It might not be helpful because the customer could have already gone to a ticketing window for help with the failed order.

SH: But what if it happens again???

Me: The solution is to make sure the ticketing server can handle the influx of requests. We can coordinate with that team. Wait. Why did you wait until the next day to resubmit orders in the admin panel? A lot of those failures happened when there were many hours left in the business day. For each order failure, an email notification is sent to the sales support email in real time. Who is monitoring that inbox? Someone must be looking at it because the sales support email is listed multiple times on the ticketing website as the technical assistance email.

SH: I know that email notification goes to the engineering team.

Me: My question is not about the engineering team. I asked who is monitoring the sales support inbox.

SH: That email … gets filtered.

Me and Product Manager: 😧🤯🤬

PM: First, you need to stop filtering that email notification. Second, your team needs to come up with a flow to handle failed orders because you told us you don’t have one. After you tried this and there’s still an issue, then we can revisit.

—-
If you’re wondering why I said no, I’m a team of one and I have a bunch of other development tasks on my plate. I’m not automating a manual task that rarely has to be performed.

At my last startup, the company decided to formally adopt core values after being in business for ten years already. They even emblazoned them on a mouse pad and sent it out to every employee as company swag. I quickly learned that those values were more for show. It was a sign of the culture going downhill. Values included teamwork and collaboration. But since I was only an IC and wasn’t a manager or director or VP, I was expected to be a mindless worker bee. Even when I just asked logical followup questions, I was treated like I was being insubordinate and “questioning their requests.”

Just trying to survive because stakeholders don’t understand what I actually have ownership for and think I have magic to fix all their websites issues.

The worst dev experience was having to interview people for job openings. I already dislike having to be the interviewee. I don’t like being the interviewer because I haven’t had a great experience with it. I’ve had a lot of people tell me what they think I want to hear instead of just answering my questions.

Surprisingly, the best was working with a recruiter for our open roles. The candidates from the recruiter were really great. Personally, I don’t have great experience with recruiters when I’m the one looking for a job. But for this case of my employer using one, it worked out. IDK if those candidates would have applied without the recruiter.

I’m most proud of my first website. Just plain html and css. It was the first time I was introduced to GitHub too. I was taking a class at the library. The teacher was the best because she showed the students how to find resources for web development and told us to don’t bother looking at the out of date workbooks. The students were cool too. It was great to be in a small class and see people of different ages learning how to code.

Sometimes I have to connect to production database and alter my dev environment so I can “log in” as a user and see what’s wrong with their account. Once in a while there is a legitimate website issue that is unique to that user’s profile. Other times it’s user error, like the user not understanding that they have to connect their membership to their online account (they think signing up for an account will connect it automatically).

I don’t like circumventing the user’s log in like this, but sometimes it’s necessary since the website is so confusing. I inherited this website, so many of the problems were formed way before I took over.

My stakeholders want a log in as user feature for website admins to use. My manager and PM don’t think that’s a good idea right now since there are over two dozen people with admin access and admin access means access to everything in the admin (there aren’t options to give permissions as needed).

The company I work for now has so much tech debt. When I find an issue, I can’t necessarily fix it right away because I have other priorities. If something isn’t a site-breaking issue, then I only fix it when a user or staff member reports it.

The website is a mess because it was built and maintained by an outside dev agency. It was so expensive to outsource that my employer decided to bring development in-house.

That’s where I came in. I found so many issues. Tech debt. UX weirdness. Newish features that no one seemed to use. It goes on.

So I’m balancing new feature development, fixing bugs, and trying to lessen our tech debt. I’m a team of one.

Welp. I think I witnessed a new job application hack. Someone listed my team’s general engineering email address for their Employee Referral.

That email address is listed publicly, but I’m pretty sure no one on my team told the applicant to list it as a referral contact. I suspect someone got the email from a Slack workspace. I had posted a job listing, in a threaded comment someone had complimented my employer’s public API, and I shared our engineering email and said we’d love to see what he builds.

It looks like someone else from that Slack saw this and decided to list the engineering email as an employee referral. I get that employee referral can mean different things to different people and it might be someone who’s new to job searching and doesn’t know better.

For my employer’s online application, an employee referral requires a name and email address for the employee. I’m curious what the applicant listed for the employee referrer’s name. Wonder if it was my name. If it is, guess I have to give my manager a heads up and tell him that I do not know this applicant.

This occurrence is a new one for me and I don’t think it’s happened to us before. And it’s not really a good tactic to get a resume read at my workplace. Where I work, my manger reviews the resumes and tells HR who he wants to set up calls with. It’s not HR or an ATS that screens resumes and sends them to my manager.

Best way to estimate a dev project for me is to establish what are the nice to haves and the must haves. A lot of times I’ll get a big list of requirements or a vague outline, so I need to figure what are our priorities.

If the project involves a new service that we have to purchase, then that project’s time estimate is one weekend because that’s how long it took other companies to implement per that service’s account manager lol 🤣

My previous employer was an e-commerce company. Most of our customers had use it or lose it funds that had to be spent by December 31 each year. So every year, the devs had to stay online until midnight on New Year’s Eve just in case there was a website issue. I didn’t witness any issue during my time there, or at least I was never contacted for support when I was on NYE duty.

They compensated by giving an extra PTO day for future use. Pre 2020, they’d allow us to leave work two hours early on NYE since the office was in NYC and getting home would be a nightmare. But you’d have to work from home to work the NYE support.

It was “optional”, but we know as a dev it’s not really optional unless you have a life and death reason not to. My first few weeks working there, my grandma had passed away. The funeral was NYE weekend so I was excused from doing the NYE support my first year because I was on bereavement leave.

The last two weeks of December were considered blackout dates for PTO, so everyone (including non devs) was not allowed to take any vacation time during those two weeks. Some people might have a problem with that if they’re into holiday celebrations and family and friend get togethers. They did observe Christmas, so that was the only day off most folks got during those two weeks. Though, the period from Thanksgiving through the end of December was stressful.

I understand some of my stakeholders have more stressful roles than others, but I really wish they’d slow down and take a moment to process.

One of them recently forwarded me a customer inquiry about an order confirmation email because the customer gave the impression that they received the email in error. The customer’s message was “2018? What is this?” It was a confirmation for an old order. A really old order. From 2018. I guess my stakeholder thought an old confirmation email was resent, but my stakeholder just had to look at the original message section of the email, which stated the email was sent in 2018. Y’all, that email was sent years before I starting working for them.

I told stakeholder that I don’t know what this customer is looking for from us. IMO since this is for an order from FOUR YEARS AGO, I don’t think we should put any more time into investigating it.

Even my Product Manager agreed that our stakeholders need to do more diligence on their end (like asking the customer why they are following up on a four year old order) before coming to Engineering and taking up our time.

I was on vacation when my employer’s new fiscal year started. My manager let me take vacation because it’s not like anything critical was going to happen. Well, joke was on us because we didn’t foresee the stupidity of others…

I had to update a few product codes in the website’s web config and deploy those changes. I was only going to be logged in for 30 minutes to complete that.

I get messaged by one of our database admins. He was doing testing and was unable to complete a payment on the website. That was strange. There was a change pushed by our offsite dev agency, but that was all frontend changes (just updating text) and wouldn’t affect payments.

We don’t want to enlist the dev agency for debugging work, especially when it’s not likely that it’s a code issue. But I was on vacation and I couldn’t stay online past the time I had budgeted for. So my employer enlists the dev agency for help. It’s going to be costly because the agency is in Lithuania, it was past their business hours, and it was emergency support.

Dev agency looks at error logs. There are Apple Pay errors, but that doesn’t explain why non Apple Pay transactions aren’t going through. They roll back my deployment and theirs, but no change. They tell my employer to contact our payment processor.

My manager and the Product Manager contact Payroll, who is the stakeholder for our payment gateways. Payroll contacts our payment gateway and finds out a service called Decision Manager was recently configured for our account. Decision Manager was declining all payments. Payroll was not the person who had Decision Manager installed and our account using this service was news to her.

Payroll works with our payment processor to get payments working again. The damage is pretty severe. Online payments were down for at least 12 hours. Our call center had logged reports from customers the night before.

At our post mortem, we had to find out who ok’d Decision Manager without telling anyone. Luckily, it was quick work. The first stakeholder up was for the Fundraising Dept. She said it wasn’t her or anyone on her team. Our VP of Analytics broke it to her that our payment processor gave us the name of the person who ok’d Decision Manager and it was someone on the Fundraising team. Fundraising then starts backtracking and says that oh yes she knew about it but transactions were still working after the Decision Manager had been configured. WTAF.

Everyone is dumbfounded by this. How could you make a big change to our payment processor and not tell anyone? How did our payment processor allow you to make this change when you’re not the account admin (you’re just a user)?

Our company head had to give an awkward speech about communication and how it’s important. The web team can’t figure out issues if you don’t tell us what you did. The company head was pissed because it was a shitty way to start off the new fiscal year. Our bill for the dev agency must have been over $1000 for debugging work that wasn’t helpful.

Amazingly, no one was fired.

This happened during the early months of WFH in the covid pandemic. I had a paired programming video interview and my interviewer had some strange behavior. IDK if he had a weird tick, but his head kept dropping to the side like he was falling asleep and he’d jerk back up again. His eyes weren’t drooping though. It kept happening throughout the interview and I was afraid he’d fall out of his chair. I wondered if he was crashing from an all nighter or his body was shutting down in some way. It was jarring enough that I wondered if I should ask the recruiter to check on my interviewer.

Web Ops Director: [looking at a screenshot of changes she had requested] This looks good. Oh by the way, revert that red color for heading text.

Me: I’m not reverting anything because there’s nothing for me to revert. I didn’t touch that text color. The website has always looked like that.

Director: [shocked pikachu face]

For those who do hiring, do you find behavioral questions to be useful?

If yes, do you prefer it when the candidate gives specific answers from their work experience? Do you use a rubric? For example, do you use the STAR (situation, task, action, response) method or something similar?

If no, why don’t you use behavioral interviewing?

I lost half my day yesterday because stakeholders made a change to one of the systems that I need. I noticed my dev environment could not longer authenticate into the system. That usually happens when there’s a “refresh” of that system. Meaning that someone copied the production instance over to the staging one, which wiped out my user credentials. One stakeholder thought he had to notify me AFTER the system refresh and not before. Another stakeholder thought it was my task to restore my user. Nope, I’m only a user for this system. I’m not responsible for any maintenance. They weren’t understanding what they had to do even after I sent them messages saying that I can no longer authenticate and I need them to check my username and password are active and correct for the staging instance.

Product Manager: Is there an event in the staging environment that we can use for testing orders?

Stakeholder: [Out of his comfort zone because he’s taking over tasks that used to belong to his assistant and he doesn’t have a new assistant yet.]There’s an event for 6/9/2022 that still has tickets available.

[Today is 8/24/2022.]

PM: You do realize that the website doesn’t allow users to buy tickets for events that are in the past?

The boss that had a positive impact on me was the one who was honest about the realities of our workplace. To some, his talks with me might come off as gossip. There’s some truth to that, but ultimately he was just doing me a favor. I think he also just needed someone to vent to since our roles were largely isolated from other colleagues. I appreciated not having the wool pulled over my eyes. He helped me understand the politics of our industry, like salaries and promotions.

Friend: What do you know about Wordpress?

Me: Why???

Friend: My assistant made changes to my organization’s website and now it’s messed up. The page formatting is off.

Me: Wordpress has a version history for some things. Maybe go back to an earlier point in time?

Friend: I think she changed something that the vendor told us not to touch.

Me: Like a custom plugin that your website vendor made?

Friend: Maybe…

Me: Why is your assistant even touching things like that?

Friend: I really don’t want to contact the vendor because I don’t think they’re very good with website development. And I have no idea what this would cost.

Me: You might have to bite the bullet on the cost. And maybe fire that assistant for a butthole move like that. At least you have messaging to explain the wonky css is due to technical difficulties. RIP to your website.

In 2020, I was working for a company that started selling at-home covid tests online. Marketing wanted us to add a survey for this product. They said this was required by the government…but we didn’t have to retain any of the answers…🤨.

Stakeholder: In user profiles, I want users to be able to renew gift memberships for their giftee.

Me: ???

SH: For example, if I buy a gift membership for you and it expires or is about to expire, then I want to be able to renew it for you.

Me: Typically, gifts aren’t the gifter’s responsibility to manage. There’s no reason for you to be able to manage my membership from your account, even if just to renew. You’re opening up Pandora’s box here. If you let users renew for giftees, you’ll eventually have a user ask if they can cancel the giftee’s membership because they got into a fight and want to stick it to the giftee.

SH: But our users aren’t using the gift membership sales flow correctly. That results in all sorts of data issues for our reporting services and we spend so much time fixing it by hand.

Me: Your sales flow is confusing. The website asks users to verify membership for a giftee in case the giftee has or had a membership. How it the gifter supposed to know that? You’re trying to make things easier for you, but you’re expecting the user to know that and comply. That’s unrealistic.

SH: But there must be a something you can do.

Me: No.

Stakeholder: Our customers aren’t getting their order confirmation emails. It’s going to their spam. Is there a way to resend the emails?

Me: Wait. The email getting sent to their spam is not the same as not getting the email. Is it both or just one of those things?

SH: Here are some emails getting sent to spam.

Me: These are sent by third party vendors that you contract with. I have no control over what they do. You need to talk to your vendors.

SH: 😮

Me: 😑