So they were having trouble with the server always being slow and maxed to 100%, so the boss told me when wait times were hitting 5+mins due to server trying to catch up, he complained at me, said if I could get the wait time to 30sec to instant he would raise my pay to 90k a year, then walked away after I agreed, I was quite serious but I don't think he thought I was, so I decided to look over the system, IDK who but they put all the calculations and processing server-side for the CA's on floor then sent the completed view to the CA, so I spent months recreating the entire system except the server only pulled the data needed then the new client would do all the processing on their computer since they weren't doing anything anyways, I did a practice run today as its one of our peak days, wait times went to barely 5secs or "instant" according to CA's, I walked into the office, slapped that hourly report down after just two hours and showed the massive increase in employees production times.

That look on his face...
That look on my face...
That look on my next check...

Bliss

My biggest dev blunder. I haven't told a single soul about this, until now.

👻👻👻👻👻👻

So, I was working as a full stack dev at a small consulting company. By this time I had about 3 years of experience and started to get pretty comfortable with my tools and the systems I worked with.

I was the person in charge of a system dealing with interactions between people in different roles. Some of this data could be sensitive in nature and users had a legal right to have data permanently removed from our system. In this case it meant remoting into the production database server and manually issuing DELETE statements against the db. Ugh.

As soon as my brain finishes processing the request to venture into that binary minefield and perform rocket surgery on that cursed database my sympathetic nervous system goes into high alert, palms sweaty. Mom's spaghetti.

Alright. Let's do this the safe way. I write the statements needed and do a test run on my machine. Works like a charm 😎

Time to get this over with. I remote into the server. I paste the code into Microsoft SQL Server Management Studio. I read through the code again and again and again. It's solid. I hit run.

....

Wait. I ran it?

....

With the IDs from my local run?

...

I stare at the confirmation message: "Nice job dude, you just deleted some stuff. Cool. See ya. - Your old pal SQL Server".

What did I just delete? What ramifications will this have? Am I sweating? My life is over. Fuck! Think, think, think.

You're a professional. Handle it like one, goddammit.

I think about doing a rollback but the server dudes are even more incompetent than me and we'd lose all the transactions that occurred after my little slip. No, that won't fly.

I do the only sensible thing: I run the statements again with the correct IDs, disconnect my remote session, and BOTTLE THAT SHIT UP FOREVER.

I tell no one. The next few days I await some kind of bug report or maybe a SWAT team. Days pass. Nothing. My anxiety slowly dissipates. That fateful day fades into oblivion and I feel confident my secret will die with me. Cool ¯\_(ツ)_/¯

*Interview*

Interviewer: We have an opening. Are you interested to work?

Me: What is that I'll be doing?

I: What technologies and languages do you know?

Me: I know Scala, Java, Spark, Angular, Typescript, blah blah. What is your tech stack?

I: Any experience working on frontend?

Me: Yes. But what do you use for it?

I: Can you work with databases?

Me: I can, on SQL based. What are yours?

I: Can you do big data processing?

Me: I know Spark, if that's what you are asking for. What is it that you actually do?

I: Any experience in cloud development?

Me: Yes. AWS? Azure? GCP?

I: Do you know CI CD?

Me: Excuse me.. I've been asking a lot of questions but you're not paying attention to what I'm asking. Can you please answer the questions I asked.

I: Yes. Go ahead.

Me: What will be my position?

I: A full stack developer.

Me: What technologies do you use in your project?

I: We use all the latest tech.

Me: Like?

I: All latest tech.

Me: You mentioned big data processing?

I: Yes. Processing data from DB and generating reports.

Me: what do you use for that?

I: Java.

Me: Are you planning to rebuild it using Spark or something and deploy in the cloud?

I: No we're not rebuilding it. Just some additions to the existing.

Me: Then what's with cloud? Why did you ask for that?

I: Just to know if you're familiar.

Me: So I'll be working with Java. Okay. What do you use for UI?

I: Flash

Me: 🙄

I sat for a couple of minutes contemplating life.

I: Are you willing to join?

Me: No. Not at all. Thankyou for the offer.

Lads, I will be real with you: some of you show absolute contempt to the actual academic study of the field.

In a previous rant from another ranter it was thrown up and about the question for finding a binary search implementation.

Asking a senior in the field of software engineering and computer science such question should be a simple answer, specifically depending on the type of job application in question. Specially if you are applying as a SENIOR.

I am tired of this strange self-learner mentality that those that have a degree or a deep grasp of these fundamental concepts are somewhat beneath you because you learned to push out a website using the New Boston tutorials on youtube. FOR every field THAT MATTERS a license or degree is hold in high regards.
"Oh I didn't go to school, shit is for suckers, but I learned how to chop people up and kinda fix it from some tutorials on youtube" <---- try that for a medical position.

"Nah it's cool, I can fix your breaks, learned how to do it by reading blogs on the internet" <--- maintenance shop

"Sure can write the controller processing code for that boing plane! Just got done with a low level tutorial on some websites! what can go wrong!"

(The same goes for military devices which in the past have actually killed mfkers in the U.S)

Just recently a series of people were sent to jail because of a bug in software. Industries NEED to make sure a mfker has aaaall of the bells and whistles needed for running and creating software.

During my masters degree, it fucking FASCINATED me how many mfkers were absolutely completely NEW to the concept of testing code, some of them with years in the field.

And I know what you are thinking "fuck you, I am fucking awesome" <--- I AM SURE YOU BLOODY WELL ARE but we live in a planet with billions of people and millions of them have fallen through the cracks into software related positions as well as complete degrees, the degree at LEAST has a SPECTACULAR barrier of entry during that intro to Algos and DS that a lot of bitches fail.

NOTE: NOT knowing the ABSTRACTIONS over the tools that we use WILL eventually bite you in the ASS because you do not fucking KNOW how these are implemented internally.

Why do you think compiler designers, kernel designers and embedded developers make the BANK they made? Because they don't know memory efficient ways of deploying a product with minimal overhead without proper data structures and algorithmic thinking? NOT EVERYTHING IS SHITTY WEB DEVELOPMENT

SO, if a mfker talks shit about a so called SENIOR for not knowing that the first mamase mamasa bloody simple as shit algorithm THROWN at you in the first 10 pages of an algo and ds book, then y'all should be offended at the mkfer saying that he is a SENIOR, because these SENIORS are the same mfkers that try to at one point in time teach other people.
These SENIORS are the same mfkers that left me a FUCKING HORRIBLE AND USELESS MESS OF SPAGHETTI CODE

Specially to most PHP developers (my main area) y'all would have been well motherfucking served in learning how not to forLoop the fuck out of tables consisting of over 50k interconnected records, WHAT THE FUCK

"LeaRniNG tHiS iS noT neeDed!!" yes IT fucking IS

being able to code a binary search (in that example) from scratch lets me know fucking EXACTLY how well your thought process is when facing a hard challenge, knowing the basemotherfucking case of a LinkedList will damn well make you understand WHAT is going on with your abstractions as to not fucking violate memory constraints, this-shit-is-important.

So, will your royal majesties at least for the sake of completeness look into a couple of very well made youtube or book tutorials concerning the topic?

You can code an entire website, fine as shit, you will get tested by my ass in terms of security and best practices, run these questions now, and it very motherfucking well be as efficient as I think it should be(I HIRE, NOT YOU, or your fucking blog posts concerning how much MY degree was not needed, oh and btw, MY degree is what made sure I was able to make SUCH decissions)

This will make a loooooooot of mfkers salty, don't worry, I will still accept you as an interview candidate, but if you think you are good enough without a degree, or better than me (has happened, told that to my face by a candidate) then get fucking ready to receive a question concerning: BASIC FUCKING COMPUTER SCIENCE TOPICS

* gays away into the night

Recipe for a Great Programmer:

Ingredients:
-Books for a computer science curriculum from a top university
-Computer
-Headphones
-Internet
-Stress ball
-Pillow
-Lighter fluid
-Food

Directions:

1. Cover computer science books with lighter fluid
2. Light books on fire
3. Use flames to cook an energy-rich meal for the thousands of hours ahead
4. Pick an IDE
5. Choose a project beyond current capabilities. Good ways to push boundaries:
- Unfamiliar domain (e.g. large scale data processing, UI programming, high performance computing, games)
- Exotic programming language
- Larger in scope than any project before
6. Shut up about your IDE
7. Attempt to build
8. Stop procrastinating on Hacker News
9. Re-attempt to build
10. Squeeze stress ball and scream into pillow as necessary to keep sanity
When stuck:
- Paste stack traces into Google
- Find appropriate mailing list to get guidance
- Realize that real learning happens when you are stuck, uncomfortable, and/or frustrated
- Seek out books, classes, or other resources AFTER you have a good understanding of your deficiencies
11. Repeat #4 to #10 for at least 10 years
12. Results guaranteed! (to the same extent static types guarantee bug-free programs)

source: nathanmarz.com

I've always elected to program a lot of my tasks instead of manually inputting them. Using a computer to solve repetitive tasks is the name of the game.

About two years ago I get roped into a something when someone was requesting an $8000 laptop to run an "program" that they wrote in Excel to pull data from our mainframe.

In reality they are using our normal application that interacts with the mainframe and screen scrapping it to populate several Excel spreadsheets.

So this guy kept saying that he needed the expensive laptop because he needed the extra RAM and processing power for his application. At the time we only supported 32 bit Windows 7 so even though I told him ten times that the OS wouldn't recognize more than 3.5 GB of RAM he kept saying that increasing the RAM would fix his problem. I also explained that even if we installed the 64 bit OS we didn't have approval for the 64 bit applications.

So we looked at the code and we found that rather than reusing the same workbook he was opening a new instance of a workbook during each iteration of his loop and then not closing or disposing of them. So he was running out of memory due to never disposing of anything.

Even better than all of that, he wanted a faster processor to speed up the processing, but he had about 5 seconds of thread sleeps in each loop so that the place he was screen scrapping from would have time to load. So it wouldn't matter how fast the processor was, in the end there were sleeps and waits in there hard coded to slow down the app. And the guy didn't understand that a faster processor wouldn't have made a difference.

The worst thing is a "dev" that thinks they know what they are doing but they don't have a clue.

What kind of supercomputer you have to use to get these fucking websites to work smoothly????

I'm on a fucking gigabit connection, ryzen 7 7700x, 32GB ram, and a fucking nvme, all it takes is opening a fucking recipe site and I'm instantly transported back to the 80s. I swear if i see another 4k asset I'm gonna punch something.

WHAT THE FUCK HAPPENED TO FUNCTION OVER FORM????

Oh do you want me to disable my addblocker??? How about: you make a site that works you fuck. No i will not fucking subscribe to your brain-dead newsletter why the fuck would I???
And since when are cookies needed for a fucking plaintext site you asshat??? Tracking??? I swear if you could you would generate metadata from my clipped fingernails if it meant you could stick "Big data" next to that zip-bomb you call a website.

I WOULD like to read your article, possibly even watch a couple of ads on my sidebar for you, but noooooo you had to have the stupid fucking google vinegrette or however the fuck they are calling the fucking thing now.

The age of the web sucks the happiness out of life, and despite having all of this processing power, I am jealous of my fathers RSS feeds.

I'm sorry web people, I know it's not your fault, I know designers and management don't give a shit how long a website takes to load. I just wanted to make a fucking omelette.

Been coding for hours, trying to automate the processing of over 500 rows of data, I refuse to do that manually.

At least I have company

Customer : c
Me : m

*Few weeks ago*
C: the server is slow, it sometimes takes 7 seconds before I see our data

(the project is 7+ years old and wasn't written by someone who is very good in SQL)

M: yeah I see that, our servers are busy with this one "process" (SQL query)
C: make it faster
M: well that's possible but it will take a few days (massive SQL spaghetti that I first have to untangle)
C: 😡 nvm then

*Yesterday*
C: server is down !
M: 🤔 *loads data from server and waits ~ 7 seconds*
M: Well what's the problem?
C: I need the data but it's so slow

WELL YOU MINDLESS IMBECILE... If something is slow it doesn't mean our god damn production server is down !
That just means that you have to give us a day or two so we can optimise the (ALSO BY YOUR REQUEST) rushed project... And save you YOUR money that YOU waste on the processing time on our server...

WIP - Scarlet Witch project
AKA me collecting/cleaning/processing data while someone else will get the glory 😂

Wanted to automate some stuff through bash scripts but did want to separate tasks so I wrote two bash scripts for data processing.

Hmm.. two scripts..... how could I even further automate thi....

*writes script which executed those two scripts*

😄

What an absolute fucking disaster of a day. Strap in, folks; it's time for a bumpy ride!

I got a whole hour of work done today. The first hour of my morning because I went to work a bit early. Then people started complaining about Jenkins jobs failing on that one Jenkins server our team has been wanting to decom for two years but management won't let us force people to move to new servers. It's a single server with over four thousand projects, some of which run massive data processing jobs that last DAYS. The server was originally set up by people who have since quit, of course, and left it behind for my team to adopt with zero documentation.

Anyway, the 500GB disk is 100% full. The memory (all 64GB of it) is fully consumed by stuck jobs. We can't track down large old files to delete because du chokes on the workspace folder with thousands of subfolders with no Ram to spare. We decide to basically take a hacksaw to it, deleting the workspace for every job not currently in progress. This of course fucked up some really poorly-designed pipelines that relied on workspaces persisting between jobs, so we had to deal with complaints about that as well.

So we get the Jenkins server up and running again just in time for AWS to have a major incident affecting EC2 instance provisioning in our primary region. People keep bugging me to fix it, I keep telling them that it's Amazon's problem to solve, they wait a few minutes and ask me to fix it again. Emails flying back and forth until that was done.

Lunch time already. But the fun isn't over yet!

I get back to my desk to find out that new hires or people who got new Mac laptops recently can't even install our toolchain, because management has started handing out M1 Macs without telling us and all our tools are compiled solely for x86_64. That took some troubleshooting to even figure out what the problem was because the only error people got from homebrew was that the formula was empty when it clearly wasn't.

After figuring out that problem (but not fully solving it yet), one team starts complaining to us about a Github problem because we manage the github org. Except it's not a github problem and I already knew this because they are a Problem Team that uses some technical authoring software with Git integration but they only have even the barest understanding of what Git actually does. Turns out it's a Git problem. An update for Git was pushed out recently that patches a big bad vulnerability and the way it was patched causes problems because they're using Git wrong (multiple users accessing the same local repo on a samba share). It's a huge vulnerability so my entire conversation with them went sort of like:

"Please don't."
"We have to."
"Fine, here's a workaround, this will allow arbitrary code execution by anyone with physical or virtual access to this computer that you have sitting in an unlocked office somewhere."
"How do I run a Git command I don't use Git."

So that dealt with, I start taking a look at our toolchain, trying to figure out if I can easily just cross-compile it to arm64 for the M1 macbooks or if it will be a more involved fix. And I find all kinds of horrendous shit left behind by the people who wrote the tools that, naturally, they left for us to adopt when they quit over a year ago. I'm talking entire functions in a tool used by hundreds of people that were put in as a joke, poorly documented functions I am still trying to puzzle out, and exactly zero comments in the code and abbreviated function names like "gars", "snh", and "jgajawwawstai".

While I'm looking into that, the person from our team who is responsible for incident communication finally gets the AWS EC2 provisioning issue reported to IT Operations, who sent out an alert to affected users that should have gone out hours earlier.

Meanwhile, according to the health dashboard in AWS, the issue had already been resolved three hours before the communication went out and the ticket remains open at this moment, as far as I know.

The day I discovered Schrödinger's lesser known paradox of simultaneously being fired and not fired.

This isn't really much of a dev story, but I figured I'd share it anyway.

About two minutes into signing into all my stuff, I suddenly was kicked out of everything. I tried logging in a few more times, and then suddenly started getting the error, "Your account has been disabled for security reasons." I couldn't sign into chat, and co-workers confirmed that I was missing from the company directory. My manager didn't come in for another two hours, and we couldn't get anyone else to answer what the hell was going on. So I was kinda panicking.

Eventually, we found out from one of our coordinators that someone else with the same name as me was leaving the company, and they had deactivated the wrong person.

It ended up getting a lot better. They told me that it could take up to 48 hours to restore my access (it took longer), so I found stuff to do so I could maintain my paycheck. One of those things was assisting someone with data collection and processing, where I eventually said, "Dude, I could totally automate this," and now that's what I'm getting paid to do.

Feel free to scroll by if you feel like it.

I am just very excited this evening because with today's commit I have reached a very important milestone in my side-project development. As of today all the [so far] 12 components are all working together and processing the main flow themselves.
No special functions, no test data in the code, nothing like that. A client is able to do its thing now as it should.

I know it doesn't sound like much, but as I'm working on this gigantic beast for 3 years now this milestone is hell of a reward for me!

Just wanted to share :)

edit: f* it! I'm getting a cake!

I've optimised so many things in my time I can't remember most of them.

Most recently, something had to be the equivalent off `"literal" LIKE column` with a million rows to compare. It would take around a second average each literal to lookup for a service that needs to be high load and low latency. This isn't an easy case to optimise, many people would consider it impossible.

It took my a couple of hours to reverse engineer the data and implement a few hundred line implementation that would look it up in 1ms average with the worst possible case being very rare and not too distant from this.

In another case there was a lookup of arbitrary time spans that most people would not bother to cache because the input parameters are too short lived and variable to make a difference. I replaced the 50000+ line application acting as a middle man between the application and database with 500 lines of code that did the look up faster and was able to implement a reasonable caching strategy. This dropped resource consumption by a minimum of factor of ten at least. Misses were cheaper and it was able to cache most cases. It also involved modifying the client library in C to stop it unnecessarily wrapping primitives in objects to the high level language which was causing it to consume excessive amounts of memory when processing huge data streams.

Another system would download a huge data set for every point of sale constantly, then parse and apply it. It had to reflect changes quickly but would download the whole dataset each time containing hundreds of thousands of rows. I whipped up a system so that a single server (barring redundancy) would download it in a loop, parse it using C which was much faster than the traditional interpreted language, then use a custom data differential format, TCP data streaming protocol, binary serialisation and LZMA compression to pipe it down to points of sale. This protocol also used versioning for catchup and differential combination for additional reduction in size. It went from being 30 seconds to a few minutes behind to using able to keep up to with in a second of changes. It was also using so much bandwidth that it would reach the limit on ADSL connections then get throttled. I looked at the traffic stats after and it dropped from dozens of terabytes a month to around a gigabyte or so a month for several hundred machines. The drop in the graphs you'd think all the machines had been turned off as that's what it looked like. It could now happily run over GPRS or 56K.

I was working on a project with a lot of data and noticed these huge tables and horrible queries. The tables were all the results of queries. Someone wrote terrible SQL then to optimise it ran it in the background with all possible variable values then store the results of joins and aggregates into new tables. On top of those tables they wrote more SQL. I wrote some new queries and query generation that wiped out thousands of lines of code immediately and operated on the original tables taking things down from 30GB and rapidly climbing to a couple GB.

Another time a piece of mathematics had to generate all possible permutations and the existing solution was factorial. I worked out how to optimise it to run n*n which believe it or not made the world of difference. Went from hardly handling anything to handling anything thrown at it. It was nice trying to get people to "freeze the system now".

I build my own frontend systems (admittedly rushed) that do what angular/react/vue aim for but with higher (maximum) performance including an in memory data base to back the UI that had layered event driven indexes and could handle referential integrity (overlay on the database only revealing items with valid integrity) or reordering and reposition events very rapidly using a custom AVL tree. You could layer indexes over it (data inheritance) that could be partial and dynamic.

So many times have I optimised things on automatic just cleaning up code normally. Hundreds, thousands of optimisations. It's what makes my clock tick.

Man, most memorable has to be the lead devops engineer from the first startup I worked at. My immediate team/friends called him Mr. DW - DW being short for Done and Working.

You see, Mr. DW was a brilliant devops engineer. He came up with excellent solutions to a lot of release, deployment, and data storage problems faced at the company (small genetics firm that ships servers with our analysis software on them). I am still very impressed by some of the solutions he came up with, and wish I had more time to study and learn about them before I left that company.

BUT - despite his brilliance, Mr. DW ALWAYS shipped broken stuff. For some reason this guy thinks that only testing a single happiest of happy path scenarios for whatever he is developing constitutes "everything will work as expected!" As soon as he said it was "done", but golly for him was it "done". By fucking God was that never the truth.

So, let me provide a basic example of how things would go:
my team: "Hey DW, we have a problem with X, can you fix this?"
DW: "Oh, sure. I bet it's a problem with <insert long explanations we don't care about we just want it fixed>"
my team: "....uhh, cool! Looking forward to the fix!"
... however long later...
DW: "OK, it's done. Here you go!"
my team: "Thanks! We'll get the fix into the processing pipelines"
... another short time later...
my team: "DW, this thing is broken. Look at all these failures"
DW: "How can that be? It was done! I tested it and it worked!"
my team: "Well, the failures say otherwise. How did you test?"
DW: "I just did <insert super basic thing>"
my team: "...... you know that's, like, not how things actually work for this part of the pipeline. right?"
DW: "..... But I thought it was XYZ?"
my team: "uhhhh, no, not even close. Can you please fix and let us know when it's done and working?"
DW: "... I'll fix it..."

And rinse and repeat the "it's done.. oh wait, it's broken" a good half dozen times on average. But, anyways, the birth of Mr. Done and Working - very often stuff was done, but rarely did it ever work!

I'm still friends with my team mates, and whenever we're talking and someone says something is done, we just have to ask if it's done AND working. We always get a laugh, sadly at the excuse of Mr. DW, but he dug his own hole in this regard.

Little cherry on top: So, the above happened with one of my friends. Mr. DW created installation media for one of our servers that was deployed in China. He tested it and "it was done!" Well, my friend flies out to China for on-site installation. He plugs the install medium in and goes for the install and it crashes and burns in a fire. Thankfully my friend knew the system well enough to be able to get everything installed and configured correctly minus the broken install media, but definitely the most insane example of "it's done!" but sure as he'll "it doesn't work!" we had from Mr. DW.

[Begin Rant] When you show your senior manager your REST Web Service and he says "Oh no nooo... I don't wanna see no code"... Me: Code?? That ain't code you fat silly fucker it's the command line output data which I spent a week parsing, batch processing, and storing into the database! [End Rant] :[

What I should do:
- organise my files
- make a pp presentation for English that I haven't even started
- learn for a test about insurances
- learn about characterising, processing and evaluating data
- more shit

What I do:
- browse devrant
- play games
- flash a new ROM until I find a good one
- restore a backup
- flash another ROM
- restore a backup
- go with the least worst ROM I had installed
- sleep
- sleep
- sleep
- sleep
- FUCKING SLEEP

At least I completely blocked YouTube...

I’m trying to add digit separators to a few amount fields. There’s actually three tickets to do this in various places, and I’m working on the last of them.

I had a nightmare debugging session earlier where literally everything would 404 unless I navigated through the site in a very roundabout way. I never did figure out the cause, but I found a viable workaround. Basically: the house doesn’t exist if you use the front door, but it’s fine if you go through the garden gate, around the back, and crawl in through the side window. After hours of debugging I eventually discovered that if I unlocked the front door with a different key, everything was fine… but nobody else has this problem?

Whatever.
Onto the problem at hand!

I’m trying to add digit separators to some values. I found a way to navigate to the page in question (more difficult than it sounds), and … I don’t know what view is rendering the page. Or what controller. Or how it generates its text.

The URL is encrypted, so I get no clues there. (Which was lead dev’s solution to having scrapeable IDs instead of just, you know, fixing them). The encryption also happens in middleware, so it’s a nightmare to work through. And it’s by the lead dev, so the code is fucking atrocious.

The view… could be one of many, and I don’t even know where they are. Or what layout. Or what partials go into building it.

All of the text on the page are “resources” — think named translations that support plus nested macros. I don’t know their names, and the bits of text I can search for are used fucking everywhere. “Confirmation number” (the most unique of them) turns up 79 matches. “Fee” showed up in 8310 places before my editor gave up looking. Really.

The table displaying the data, which is what I actually care about, isn’t built in JS or markup, but is likely a resource that goes through heavy processing. It gets generated in a controller somewhere (I don’t know the resource name so I can’t find it), and passed through several layers of “dynamic form” abstraction, eventually turned into markup, and rendered as a partial template. At least, that’s how it worked in the previous ticket. I found a resource that looks right, and there’s only the one. I found the nested macros it uses for the amount and total, and added the separators there… only to find that it doesn’t work.

Fucking dead end.

And i have absolutely nothing else to go on.

Page title? “Show”
URL? /~LiolV8N8KrIgaozEgLv93s…
Text? All from macros with unknown names. Can’t really search for it without considerable effort.
Table? Doesn’t work.
Text in the table? doesn’t turn up anything new.
Legal agreement? There are multiple, used in many places, generates them dynamically via (of course) resources, and even looking through the method usages, doesn’t narrow it down very much.

Just.
What the fuck?
Why does this need to be so fucking complicated?

And what genius decided “$100000.00” doesn’t need separators? Right, the lot of them because separators aren’t used ANYWHERE but in code I authored. Like, really? This is fintech. You’d think they would be ubiquitous.

And the sheer amount of abstraction?
Stupid stupid stupid stupid stupid.

It's 17:55... Did much work that day since I came in earlier than usual, so I could leave in time and do some shopping with the girlfriend.

A colleague comes in to my room, a tad distressed. He had accidentally ran a fixture script on a production environment database (processing a shipload of records per minute), truncating all tables...

Using AWS RDS to rollback the transaction log takes up about 20m. I had to do that about 5 times to estimate the date and time of when the fixture script ran... Since there was no clear point in time...

Finally I get to the best state of the data I could get. I log in remotely run some queries. All is well again... With minor losses in data.

I try to download a dump using pg_dump and apparently my version is mismatched with the server. I add the latest version to aptitudes source list of postgres repo and I am ready to remove and purge the current postgres client and extensions...

sudo apt-get remove post*
Are you sure? (Y/n) *presses enter and enters into a world of pain*

Apparently a lot of system critical applications start with post... T_T

!rant

Did you ever have that feeling of "what the fuck - how did I do THAT" - I just had one; I was fine with the algorithm taking like 15/25 minutes to fetch, process, save all data, now it shreds through 10 times of the previous amount in like 7 minutes.

This was about 3 years ago. I’m on vacation and just getting off the plane, when my boss calls me on his cellphone. Apparently the crontab on our main file upload server had gotten nuked, and he was asking if there were any backups.

A word about this server. I work with video, so this thing is doing about a few gigabits of traffic incoming at any moment. The cron jobs are necessary to move and organize these massive files into a sane scheme for processing. Hundreds of drop folders receiving thousands of files resulting in terabytes of data every single day. Our storage vendor tells us we have the third largest deployment they know about.

No cron jobs mean all of this content is just sitting around piling up. I tell him sorry, try contacting $otherAdmin since he’s more familiar with that system.

A few days later, after the vacation, I come back in. $boss and $otherAdmin have reconstructed the crontab from scratch after an all nighter.

I ask how it got deleted.

$boss was training some people how to set up new customers on this file server, and he told the trainees to open the crontab in read-only mode. One of them ran:

crontab -r

Yes, we back up our crontabs now.

So I joined this financial institution back in Nov. Selling themselves as looking for a developer to code micro-services for a Spring based project and deploying on Cloud. I packed my stuff, drove and moved to the big city 3500 km away. New start in life I thought!
Turns out that micro-services code is an old outdated 20 year old JBoss code, that was ported over to Spring 10 years ago, then let to rot and fester into a giant undocumented Spaghetti code. Microservices? Forget about that. And whats worse? This code is responsible for processing thousands of transactions every month and is currently deployed in PROD. Now its your responsibility and now you have to get new features complied on the damn thing. Whats even worse? They made 4 replicas of that project with different functionalities and now you're responsible for all. Ma'am, this project needs serious refactoring, if not a total redesign/build. Nope! Not doing this! Now go work at it.
It took me 2-3 months just to wrap my mind around this thing and implement some form of working unit tests. I have to work on all that code base by myself and deliver all by myself! naturally, I was delayed in my delivery but I finally managed to deliver.
Time for relief I thought! I wont be looking at this for a while. So they assign me the next project: Automate environment sync between PROD and QA server that is manually done so far. Easy beans right? And surely enough, the automation process is simple and straightforward...except it isnt! Why? Because I am not allowed access to the user Ids and 3rd party software used in the sync process. Database and Data WareHouse data manipulation part is same story too. I ask for access and I get denied over and over again. I try to think of workarounds and I managed to do two using jenkins pipeline and local scripts. But those processes that need 3rd party software access? I cannot do anything! How am I supposed to automate job schedule import on autosys when I DONT HAVE ACCESS!! But noo! I must think of plan B! There is no plan B! Rather than thinking of workarounds, how about getting your access privileges right and get it right the first time!!
They pay relatively well but damn, you will lose your sanity as a programmer.
God, oh god, please bless me with a better job soon so I can escape this programming hell hole.
I will never work in finance again. I don't recommend it, unless you're on the tail end of your career and you want something stable & don't give a damn about proper software engineering principles anymore.

Putty remote executuon vulnerability(no patch yet)

The vulnerability allows a remote attacker to execute arbitrary code on the target system.

The vulnerability exists due to unspecified input validation error when processing data, received from SSH server. A remote attacker can trick the victim to connect to a specially crafted SSH server and execute arbitrary code on the target system with privileges of the current user.

Successful exploitation of the vulnerability may allow an attacker to compromise vulnerable system.

What you are expected to learn in 3 years:

power electronics,
analogue signal,
digital signal processing,
VDHL development,
VLSI debelopment,
antenna design,
optical communication,
networking,
digital storage,
electromagnetic,
ARM ISA,
x86 ISA,
signal and control system,
robotics,
computer vision,
NLP, data algorithm,
Java, C++, Python,
javascript frameworks,
ASP.NET web development,
cloud computing,
computer security ,
Information coding,
ethical hacking,
statistics,
machine learning,
data mining,
data analysis,
cloud computing,
Matlab,
Android app development,
IOS app development,
Computer architecture,
Computer network,
discrete structure,
3D game development,
operating system,
introduction to DevOps,
how-to -fix- computer,
system administration,
Project of being entrepreneur,

and 24 random unrelated subjects of your choices

This is a major called "computer engineering"

More adventures of working with data scientists:

Doing combinatorial analysis.

One of the post processing scripts throws away combinatorial information, analyses significance of individual data points instead.

Code was so complicated, nobody noticed.

I write a thesis about some data mining project.
I need to process 15 million tracking points - today I finally finished coding my algorithm. I was pretty excited and pressed run:
Now, after 2,5 hours of processing, my program got already through a third of all tracking points, as I realized:
I gave my algorithm a non-valid output path.
And that thouht stuck me, as I was already on my way home.
Now I have to go tomorrow (on my day off) to work, to fix and run it again.

Our company has internal webpage to request software, be it freeware or licensed.
Today, I found there "Software engineering bundle" designated for "software developers and data scientists who require advanced compute and data processing tools".
The software bundle contains PuTTY, 7-zip and Notepad++.

Aaaaaaaargh!! Fing ashole!!
I got a major blocker reported, tried to connect to client, two of the user accounts were locked out because some genious used the last months password too many times.. FUUUU!! This happens almost every month!! FU! I go to the support dpt to check WTH is with those user accounts and got told the VPN is fucked up anyway so I will not be able to connect in any casr (disconnecting, bad transfer rate, it has a flue or prebirth cramps...whatever...). Ok, I ask if anyone notified our network admins and theirs.. And in response one guy mumbles something... I asked really really pissed off (due to the seriousnrs of the situation, we have max 8h to fix blockers and must check what is going on in minutes) if he is talking to me and answering my question or just talking to himself. He then a little bit more audiably said: we all are unable to work, you are not the only one with this problem & if you have a solutio... I already stormed out. Yes, everyone has problems connecting, no not everyone has a fucking blocker assigned to them!! Mayor malfunction on our system is not the same as archiving old processing data!!!

Simple yes or no question: did anyone notify our network admins & client's network admins?! And client's management that we have technical problems and cannot check the blocker situation immediately?! And I get a mumbling incompetents guy response... OmFG yes, I have a solution for you!! Go and jump of of the terrace!!

So I just started a part time job in a hospital research center - because the processing is long I got a temporary user name and password (that belong to the main HR secretary) so I can start work straight away (mainly data analytics)
The kick?
Administrator privileges.
I can access edit create or delete everything in the entire fucking database. On my first God damn day.
In the 2nd largest hospital in the fucking country.
Agh. How do systems survive with so many dumb security breaches?

The company considers the project manager I work with to be the best. After working with him, I consider him to be everything that is wrong with project management.

This PM injects himself into everything and has a way of completely over-complicating the smallest of things. I will give an example:

We needed to receive around 1000 rows of data from our vendor, process each row, and host an endpoint with the data in json. This was a pretty simple task until the PM got involved and over complicated the shit out of it. He asks me what file format I need to receive the data. I say it doesnt really matter, if the vendor has the data in Excel, I can use that. After an hour long conversation about his concerns using Excel he decides CSV is better. I tell him not a problem for me, CSV works just as good. The PM then has multiple conversations with the Vendor about the specific format he wants it in. Everything seems good. The he calls me and asks how am I going to host the JSON endpoints. I tell him because its static data, I was probably going to simply convert each record into its own file and use `nginx`. He is concerned about how I would process each record into its own file. I then suggest I could use a database that stores the data and have an API endpoint that will retrieve and convert into JSON. He is concerned about the complexities of adding a database and unnecessary overhead of re-processing records every time someone hits the endpoint. No decision is made and two hours are wasted. Next day he tells me he figured out a solution, we should process each record into its own JSON file and host with `nginx`. Literally the first thing I said. I tell him great, I will do that.

Fast forward a few days and its time to receive the payload of 1000 records from the Vendor. I receive the file open it up. While they sent it in CSV format the headers and column order are different. I quietly without telling the PM, adjust my code to fit what I received, ran my unit test to make sure it processed correctly, and outputted each record into its own json file. Job is now done and the project manager gets credit for getting everything to work on the first try.

This is absolutely ridiculous, the PM has an absurd 120 hours to this task! Because of all the meetings, constant interruptions, and changing of his mind, I have 35 hours to this task. In reality the actual time I spent writing code was probably 2-3 hours and all the rest was dealing with this PM's meetings and questions and indecisiveness. From a higher level, he appears to be a great PM because of all the hours he logs but in reality he takes the easiest of tasks and turns them into a nightmare. This project could have easily been worked out between me and vendor in a 30 min conversation but this PM makes it his business to insert himself into everything. And then he has the nerve to complain that he is so overwhelmed with all the stuff going on. It drives me crazy because this inefficacy and unwanted help makes everything he touches turn into a logistical nightmare but yet he is viewed as one of the companies top Project Managers.

After you automate away task into a tool or program, do you tell your boss or keep it to yourself so they remain amazed at your efficiency and productivity?

And for data analysis ones, that you have godly data processing skills as you can somehow read a 1,000,000 line log file and find the root cause and other insights in an hour or minutes?

A server application pulled off some sort of listings as table. Problem was, it crashed with some thousand data files after one and a half hours. I looked into that, and couldn't stop WTFing.

A stupid server side script fetched the data in XML (WTF!) and then inserted shit node-wise (WTF!!), which was O(n^2) - in PHP and on XML! Then it converted the whole shebang into HTML for browser display although users would finally copy/paste the result into Excel anyway.

The original developer even had written a note on the application page that pulling the data "could take long". Yeah because it's so fucking STUPID that Clippy is an Einstein in comparison, that's why!

So I pulled the raw data via batch file without XML wrapping and wrote a little C program for merging the dumped stuff client-side in O(n), spitting out a final CSV for Excel import.

Instead of fucking the server for 1.5 hours and then crashing, shit is done after 7 seconds, out of which the actual data processing takes 40 bloody milliseconds!

A story from around 2005:

Customer laying out specifications: “We expect this software to need to last 25 years or so, and it will need to keep historical file processing data by dates for at least that long, assume storage is no issue.”

Devs at the time: “look best I can do is support that start with 200 or 201, anything else is really too much to ask. Also understanding how to work with dates at all and not just string manipulation is waaaaayyy hard yo.”

Fuck you lazy motherfuckers. This is why people thought Y2K would be a problem.

TechSuppDept: There is a critical antivirus upgrade we need to do...

Me(email): can you give me 48 more hours, ive already spent 7days processing my data... cpu, mem already at 97% utility

TechSuppDept: Please explain your email.

Me: .

Never lose your sense of wonder when it comes to working with clients. Client berated us saying her data were outdated. Ok. Check the file the third party that generated the data is sending us.

Outline all stated discrepancies in the data back to the client, showing that everything lines up with what we are receiving.

Client is frustrated. Contact the third party in their behalf.

Third party support: “oh yea, client had us start sending data to your competitor like a month ago”

Bruh. Bruh. Bruh.

Fortunately the client wants to stay with us and is getting their data pointed back but how in the hell do you forget that. The reason the client when looking at competition (at least guessing looking at previous call records) is to get faster processing of the data coming from the third party. How are you gonna forget you turned off the sending when you are so worried about speed?! Most of our clients are running 7-8 figure businesses by the way.

Ok, here goes...

I was once asked to evaluate upgrade options for an online shop platform.

The thing was built on Zend 1, but that's not the problem.

The geniuses that worked on it before didn't have any clue about best practices, framework convention, modular thinking, testing, security issues...nothing!

There were some instances when querying was done using a rudimentary excuse for a model layer. Other times, they would just use raw queries and just ignore the previous method. Sometimes the database calls were made in strange function calls inside randomly loaded PHP files from different folders from all over the place. Sometimes they used JOINs to get the data from multiple tables, sometimes they would do a bunch of single table queries and just loop every data set to format it using multiple for loops.

And, best of all, there were some parts of the app that would just ignore any ideea of frameworks, conventions and all that and would be just a huge PHP file full of spagetti code just spalshed around, sometimes with no apparent logic to it. Queries, processing, HTML...everything crammed in one file...

The most amazing thing was that this code base somehow managed to function in production for more than 5 years and people actualy used it...

Imagine the reaction I got from the client the moment I said we should burn it to the ground and rebuild the whole thing from scratch...

Good thing my boss trusted me and backed me up (he is a great guy by the way) and we never had to go along with that Frankenstein monster...

I fixed my big data processing code, I think. If all works as planned, I'll wake up to some processed data that I can do some statistical analysis on... I hope...

I had a wonderful run-in with corporate security at a credit card processing company last year (I won't name them this time).
I was asked design an application that allowed users in a secure room to receive instructions for putting gift cards into envelopes, print labels and send the envelopes to the post. There were all sorts of rules about what combinations of cards could go in which envelopes etc etc, but that wasn't the hard part.
These folks had a dedicated label printer for printing the address labels, in their secure room.
The address data was in a database in the server room.
On separate networks.
And there was absolutely no way that the corporate security folks would let an application that had access to a printer that was on a different network also have access to the address data.
So I took a look at the legacy application to see what they did, to hopefully use as a precedent.
They had an unsecured web page (no, not an API, a web page) that listed the addresses to be printed. And a Windows application running on the users' PC that was quietly scraping that page to print the labels.
Luckily, it ceased to be an issue for me, as the whole IT department suddenly got outsourced to India, so it became some Indian's problem to solve.

Hiring a third party to help us with something...
Third party: yeah okay, we know what we need. Can we get access to your git repo
Me: sure, I'll make sure you'll get it
(To the admins): hey can you get them access to our git server?
Admins: did they sign the personal data processing contract?
Me: oh they won't work with any personal data. It's a dev server and they only need access to the source code. And the usual contracts and NDAs are already done
Admins: well we still need the other one.

... Sure. Why not. Just delays the start of the process for... Like a week and a half until that useless bit of paper has passed through all the necessary departments. Not like time's an issue. Right?

Was talking to a cousin of mine who's a priest and who spent 10 years in the Vatican, apparently they have 2-3 servers devoted specifically to acting as a firewall processing the data coming in and out of its network and it's a continuous stream of people trying to hack it, like think it would be pretty cool to see what way they have to whole thing set up, might see if he could take me on a tour (might even get a look at the private archives, though apparently it's mainly boring letters about popes meeting their mother and stuff

I have recently come into some spare time and I decided to build a game.
This is my first time building a game, mostly just worked on IoT and data processing, and I need tips on how to avoid becoming addicted to working on this hobby.

Many years ago, when I moved from a semi-experienced developer to an absolute beginner project manager at another company, my very first project was an absolute clusterfuck.

The customer basically wanted to scrape signups to their EventBrite events into their CRM system. The fuckery began before the project even started, when I was told my management that we HAD to use BizTalk. It didn't matter that we had zero experience with BizTalk, or that using BizTalk for this particular project was like using a stealth bomber to go down to the shops for a bottle of tequila (that's one for fans of Last Man on Earth). It's designed to be used by an experienced team of developers, not a small inexperienced 1-person dev team I had. The reason was for bullshit political reasons which I wasn't really made clear on (I suspect that our sales team sold it to them for a bazillion pounds, and they weren't using it for anything, so we had to justify us selling it to them by doing SOMETHING with it). And because this was literally my first project, I was young and not confident at all, and I wanted to be the guy who just got shit done, I didn't argue.

Inevitably, the project was a turd. It went waaay over budget and time, and didn't work very well. I remember one morning on my way to work seriously considering ploughing my car into a ditch, so that I had a good excuse not to go into work and face that bullshit project.

The good thing is that I learned a lot from that. I decided that kind of fuckery was never going to happen again.

A few months later I had an initial meeting with a potential customer (who I was told would be a great customer to have for bullshit political reasons) - I forget the details but they essentially wanted to build a platform for academic researchers to store data, process it using data processing plugins which they could buy, and commersialise it somehow. There were so many reasons why this was a terrible idea, but when they said that they were dead set on using SharePoint (SharePoint!!!) as the base of the platform, I remembered my first project and what happened.

I politely explained my technical and business concerns over the idea, and reasons why SharePoint was not a good fit (with diagrams and everything), suggested a completely different technology stack, and scheduled another meeting so they could absorb what I had said and revisit. I went to my sales and head of development and basically told them to run. Run fast, and run far, because it won't work, these guys are having some kind of fever dream, it's a clusterfuck in the making, and for some reason they won't consider not using SP.

I never heard from them again, so I assume we dropped them as a potential client. It felt amazing. I think that was the single best thing I did for that company.

Moral of the story: when technology decisions are made which you know are wrong, don't be afraid to stand up and explain why.

aaaaaghh fucking Handlers man. Android is so fucking full of shit, i wonder why am i still doing it. love is pain.

Why can't there be one mother fucking solution to all lazy ass asynchronous programming? handlers, threadpools, asynctask, executers, Broadcasts, intentService, coroutines, rxjava,.... i don't what new stuff are people snorting these days.

Ok , leave everything. A handler is class- no sorry, Handler, alongside some fucking Looper clss (and maybe some more stuff i don't know) other classes is a way of handling inter thread communication. Handlers can:

-send data to ui thread
-recieve data from ui thread
-send "messages" to ui thread
-recieve "messages" from ui thread.
- can be attached to ui thread
- can be attached to any child thread
- can be accessed anonymosly via any view
- can be present in multiple places, working together
- can kill night king with a dagger
- can do porn better than johnny sins
- can run for president of the whole fucking world
- do some more shits that i have yet to discover

And where do i find this? buried deep insides some medium articles or in some guy's horrible accent video.
Is background processing really this much of a toughnut to crack?
earlier i was all about using asynctask or foreground/background services, because these are the most easy to understand abstraction of a fairly difficult topic.
But as i see more projects, i see underlying apis like handlers, threadpools , executers , being directly used.
Why cant there be a fucking single abstraction, that could be "lightly tweaked" to handle every ugly case.

During one of our visits at Konza City, Machakos county in Kenya, my team and I encountered a big problem accessing to viable water. Most times we enquired for water, we were handed a bottle of bought water. This for a day or few days would be affordable for some, but for a lifetime of a middle income person, it will be way too much expensive. Of ten people we encountered 8 complained of a proper mechanism to access to viable water. This to us was a very demanding problem, that needed to be sorted out immediately. Majority of the people were unable to conduct income generating activities such as farming because of the nature of the kind of water and its scarcity as well.
Such a scenario demands for an immediate way to solve this problem. Various ways have been put into practice to ensure sustainability of water conservation and management. However most of them have been futile on the aspect of sustainability. As part of our research we also considered to check out of the formal mechanisms put in place to ensure proper acquisition of water, and one of them we saw was tree planting, which was not sustainable at all, also some few piped water was being transported very long distances from the destinations, this however did not solve the immediate needs of the people.We found out that the area has a large body mass of salty water which was not viable for them to conduct any constructive activity. This was hint enough to help us find a way to curb this demanding challenge. Presence of salty water was the first step of our solution.

SOLUTION

We came up with an IOT based system to help curb this problem. Our system entails purification of the salty water through electrolysis, the device is places at an area where the body mass of water is located, it drills for a suitable depth and allow the salty water to flow into it. Various sets of tanks and valves are situated next to it, these tanks acts as to contain the salty water temporarily. A high power source is then connected to each tank, this enable the separation of Chlorine ions from Hydrogen Ions by electrolysis through electrolysis, salt is then separated and allowed to flow from the lower chamber of the tanks, allowing clean water to from to the preceding tanks, the preceding tanks contains various chemicals to remove any remaining impurities. The whole entire process is managed by the action of sensors. Water alkalinity, turbidity and ph are monitored and relayed onto a mobile phone, this then follows a predictive analysis of the data history stored then makes up a decision to increase flow of water in the valves or to decrease its flow. This being a hot prone area, we opted to maximize harnessing of power through solar power, this power availability is almost perfect to provide us with at least 440V constant supply to facilitate faster electrolysis of the salty water.
Being a drought prone area, it was key that the outlet water should be cold and comfortable for consumers to use, so we also coupled our output chamber with cooling tanks, these tanks are managed via our mobile application, the information relayed from it in terms of temperature and humidity are sent to it. This information is key in helping us produce water at optimum states, enabling us to fully manage supply and input of the water from the water bodies.
By the use of natural language processing, we are able to automatically control flow and feeing of the valves to and fro using Voice, one could say “The output water is too hot”, and the system would respond by increasing the speed of the fans and making the tanks provide very cold water. Additional to this system, we have prepared short video tutorials and documents enlighting people on how to conserve water and maintain the optimum state of the green economy.
IBM/OPEN SOURCE TECHNOLOGIES
For a start, we have implemented our project using esp8266 microcontrollers, sensors, transducers and low payload containers to demonstrate our project. Previously we have used Google’s firebase cloud platform to ensure realtimeness of data to-and-fro relay to the mobile. This has proven workable for most cases, whether on a small scale or large scale, however we meet challenges such as change in the fingerprint keys that renders our device not workable, we intend to overcome this problem by moving to IBM bluemix platform.
We use C++ Programming language for our microcontrollers and sensor communication, in some cases we use Python programming language to process neuro-networks for our microcontrollers.

Any feedback conserning this project please?

So... Yesterday I ordered a meal and it had whole jalapenos in it. I didn't order jalapeños. I love the taste but I hate toilet visits after. Hence, was putting them aside. But then I got into that new code, jumping around this new project I'll be working on. We were getting intimate. I liked the architecture, I liked it a lot - it was using event sourcing and respected CQRS. Suddenly I realised I ate everything. Including jalapeños. And the only reason I noticed is because I was eating with my hands. And my eye got watery. And I wiped it.

So, yeah. Yesterday for the first time in my life I was pouring milk into my eyes. Does this count as a proper dev rant? I don't know. Fuck the protein interface that can't process simple food orders, though.

Well it's a bit long but worth reading, two crazy stories in one rant:

So there are 2 things to consider as being my first job. If entrepreneurship counts, when I was 16 my developer friend and I created a small local music magazine website. We had 2 editors and 12 writers, all music enthusiasts of more or less our age. We used a CMS to let them add the content. We used a non-profit organization mentorship and got us a mentor which already had his exit, and was close to his next one. The guy was purely a genius, he taught us all about business plans, advertising, SEO, no-pay model for the young journalists (we promised to give formal journalist certificates and salary when the site grows up)
We hired a designer, we hired a flash expert to make some advertising campaigns and started filling the site with content.

Due to our programming enthusiasm we added to the raw CMS some really cool automation: We scanned our country's radio charts each week using a cron job and the charts' RSS, made a bot to search the songs on youtube and posted the first search result as an embedded video using some reg-exps. This was one of the most fun coding times I've had. Doing these crazy stuff with none to little prior knowledge really proved me I can do anything with the power of will.

Then my partner travelled to work in an internship in the Netherlands and I was too lazy to continue it on my own and it closed, not so surprisingly for a 16 years old slacker boy.

Then the mentor offered my real first job. He had a huge forum (14GB of historical SQL) but it was dying, the CMS version was very old and he wanted me to upgrade it to the latest. It didn't seem hard at first, because there were very clear instructions in the CMS website on how to do that. However, the automation upgrade scripts didn't work well because the forum owners added some raw code (not MVC plugins but bad undocumented code) and some columns to the SQL tables. I didn't give up and decided to migrate between the versions without the scripts. I opened a new CMS and started learning by heart all of the database columns so I can make a script to migrate between the versions. The first tests ran forever because processing 14GB of data on a single home computer is not a task meant to be done. I didn't give up. I made an old forum and compared the table structures and code with my mentor's. I think I didn't exhaustively finish this solution, the task was too big on my shoulders and eventually I gave up. I still owe thanks for that mentor for teaching me how to bare with seemingly (and practically) impossible tasks, for learning not to fear from being a leader and an entrepreneur and also for paying me in time even though I didn't deliver anything 😂

I need to encrypt some large files at rest and then decrypt them immediately prior to processing.

App and files are on a Linux system (CentOS). App is in C. Machine is controlled by a third party.

What encryption libraries would you recommend? And, is there any clever way of managing the decryption key beyond compiling it in the code and doing some basic obfuscation?

Are they fancy obfuscation libraries out there, for example?

And, the reason I'm not going to SO (well, one reason) is that I don't want to have 50 answers that tell me that's it's impossible to 100% protect data on a machine you don't control. This I understand---just looking for "best effort" solution.

The worst part of being a dev? Working in teams.

And I don't mean that in the "I'm the best ninja code wizard in the whole world and you're all holding me back" kinda way. I'm thinking more in the lines of someone who has to deal with that kind of attitude on a daily basis. As someone who recently was put in a leading position in a dev team, this is by far one of the worst experiences that came with it.

Some examples?
- One dev completely changed the naming scheme for variables in a class he worked on for one. single. bug fix. His reason? He just didn't like it!
- Another one noticed that data he was supplied with was not in the specified format. Instead of flagging this with the project leads, he just rewrote his parser to fit the data. A couple of weeks later the supplier noticed the error, fixed the format and suddenly everyone wondered why the software failed processing the data.
- Or that one senior dev, that just refuses to accept changes because "it was always done like this and it worked" No, it didn't. That's why it was changed!

Once a dev team reaches a certain size, people need to realize that stuff like coding rules and process guidelines are not there to annoy them but to help the whole team work as efficient as possible. I don't care how good a programmer you are, if you can't check your ego you don't belong in any kind of team-oriented development project!

For shit's sake, data stream processing really is only for people with high throughput looking to do transformations on their data; not for people aggregating <10Gb/day of data.

Fuck me DSP is going to be the new buzzword of 2020 and I'm not looking forward to it. I've already got stakeholders wondering if we can integrate it when we dont have the need, nor the resources or funds.

AI here, AI there, AI everywhere.
AI-based ads
AI-based anomaly detection
AI-based chatbots
AI-based database optimization (AlloyDB)
AI-based monitoring
AI-based blowjobs
AI-based malware
AI-based antimalware
AI-based <anything>
...

But why?
It's a genuine question. Do we really need AI in all those areas? And is AI better than a static ruleset?

I'm not much into AI/ML (I'm a paranoic sceptic) but the way I understand it, the quality of AI operation correctness relies solely on the data it's
datamodel has been trained on. And if it's a rolling datamodel, i.e. if it's training (getting feedback) while it's LIVE, its correctness depends on how good the feedback is.

The way I see it, AI/ML are very good and useful in processing enormous amounts of data to establish its own "understanding" of the matter. But if the data is incorrect or the feedback is incorrect, the AI will learn it wrong and make false assumptions/claims.

So here I am, asking you, the wiser people, AI-savvy lads, to enlighten me with your wisdom and explain to me, is AI/ML really that much needed in all those areas, or is it simpler, cheaper and perhaps more reliable to do it the old-fashioned way, i.e. preprogramming a set of static rules (perhaps with dynamic thresholds) to process the data with?

There are a couple:

A system that updates user accounts to connect them into our wifi system by parsing thousands of processing files written in Clojure. The project was short lived and mainly experimental, It has complete test cases and the jar generated from it is still purring silently on the main application. It was used to replace an $85k vendor application that made no fucking sense. The code has not been touched in 2 years and the jar is still there. The dba mentioned the solution to the vendor, the vendor tried buying it from me, but being that it belongs to the institution nothing was touched, still, it got the VP's attention that I can make programs that would be bought for that level, it caught his attention even more when I showed him the codebase and he recognized a Lisp variant (he is old, and was back in the day a Fortran and Cobol developer)

A small Python categorical ML program that determines certain attributes of user generated data and effectively places them on the proper categories on the main DB. The program generates estimates of the users and the predictions have a 95% correctness rate. The DBA still needs to double check the generated results before doing the db updates. I don't remember how I coded it because I was mostly drunk when I experiment on the scenario. It also got the attention of the VP and director since the web tech manager was apparently doing crazy ML shit that they were not expecting me to do, it made them paranoid that I would eventually leave for a ML role somewhere, still here, but I want more moneys!!

A program that generates PDF documentation from user data, written in Go, Python and Perl (yes Perl) I even got shit from the lead developer since I used languages outside of their current scope of work. Dude had no option but to follow along with it :P since I am his boss

Many more. I am normally proud of my work code. But my biggest moment is my current ntural language processing unit that I am trying to code for my home, but I don't have enough power to build it with my computers, currently, my AI is too stupid, but sometimes it does reply back to my commands and does the things I ask it to do (simple things, opening a browser, search for a song etc) but 7 times out of ten it wont work :P

spend better part of my day processing about 400M rows of data with MySql, woo ain't no single line of code that can do that in 12hrs 😊

Learning Image Processing,Deep Learning,Machine Learning,Data modeling,mining and etc related to and also work on them are so much easier than installing requiremnts, packages and tools related to them!

We specified a very optimistic setup for a data science platform for a client....

Minimum one machine with a 16 core CPU with 64GB RAM to process data.....

Client's IT department: Best we can do is an 8 core 16GB server.

Literally what I have on my laptop.

Data scientist doesn't use any out-of-memory data processing framework, e.g. Dask, despite telling him it's the best way to be economical on memory; ipykernel kills the computation anyway because it runs out of memory.

Data scientist has a 64GB machine himself so he says it's fine.

Purpose of the server: rendered pointless.

Junior Software Developer Job( $37k-$42k USD)

-1 year experience
- J2EE, Javascript, HTML, XML, SQL
- object oriented design and implementation
- management of relational and non-relational such as Oracle, PostGreSQL and Cassandra
- Lifecycle and Agile methods
- Familiarity with the Eclipse development environment and with tools such as Hibernate, JMS, ,TomCat/Gemini/Jetty, OSGi.
• UNIX skills, including Bash or other scripting language
• Experience installing and configuring software packages
• ActiveMQ troubleshooting/knowledge
• Experience in scientific data processing and analytical science in general
• Automated testing tools and procedures, including JUnit testing, Selenium, etc.
• Experience in interfacing with scientific instrumentation, potentially over IP networks
• Familiarity with modern web development, user interface and other ever-evolving front-end
technologies, such as React, TypeScript, Material, Jest, etc.

I am betting they don't get many people applying.

I am trying to "invent" secure client-side authentication where all data are stored in browser encrypted and only accessible with the correct password. My question is, what is your opinion about my idea. If you think it is not secure or there is possible backdoor, let me know.

// INPUT:
- test string (hidden, random, random length)
- password
- password again
// THEN:
- hash test string with sha-512
- encrypt test string with password
- save hash of test string
// AUTH:
- decrypt test string
- hash decrypted string with sha-512
- compare hashes
- create password hash sha-512 (and delete password from memory, so you cannot get it somehow - possible hole here because hash is reversible with brute force)

// DATA PROCESSING
- encrypt/decrypt with password hash as secret (AES-256)

Thanks!

EDIT: Maybe some salt for test string would be nice

The best moments are when you've been struggling with an implementation for a few days, and then things start to work. I had this happen last week. I have a Windows desktop app processing product dimensional data from multiple warehouses, then sending that data across the country and transposing into a data lake, joining several databases, and sending detailed reports. It was a struggle from start to finish, with lots of permissions issues, use cases to consider, and data accuracy. Finally, I break through and when I step back, I get to see this well-oiled machine of conjoined ideas run through to its eloquent, seemingly fleeting, conclusion. That feeling you get that makes you throw your hands in the air for a job well done! It's very exciting.

Had my first programming encounter when I was 12 at school with Turbo Pascal and VB, I was the best in my class but didn't really got struck by it.

It wasn't until I was 19 that I discovered Arduino and Processing, started learning C++ and Java and decided to switch from Electronic Engineering to Computer Engineering.

Since I was into music and used to make guitar pedals, the first things I programmed were a bunch of audio effects with Pure Data and some controllers with Arduino since I wanted to make a digital pedal with a raspberry, but as usual I never completed it ¯\_(ツ)_/¯

Working for a large client converting paper forms to the web. Stated goals, simplify data entry for clients, improve data quality, reduce resourcing in backend human processing.

We met to review prototype and discuss workflow questions. Crazy deadlines, with the usual changing scope creep.

We start to point out the need for data validation, to shorten # of questions based on answers.

Business says no. All forms should be submittable regardless of what user enters, don’t put validations in because all that warning messaging confuses them and takes up more time.

Web form should behave like the paper copy....

Welcome to 1975!!! This is why 2018 won’t be like 2018...

My work product: Or why I learned to get twitchy around Java...

I maintain a Java based test system, that tests a raster image processor. The client is a Java swing project that contains CORBA bindings to the internal API of the raster image processor. It also has custom written UI elements and duplicated functionality that became available in later versions of Java, but because some of the third party tools we use don't work with later versions of Java for some reason, it's not possible to upgrade Java to gain things as simple as recursive directory deletion, yes the version of Java we have to use does not support something as simple as that and custom code had to be written to support it.

Because of the requirement to build the API bindings along with the client the whole application must be built with the raster image processor build chain, which is a heavily customised jam build system. So an ant task calls out to execute a jam task and jam does about 90% of the heavy lifting.

In addition to the Java code there's code for interpreting PostScript files, as these can be used to alter the behaviour of the raster image processor during testing.

As if that weren't enough, there's a beanshell interface to allow users to script the test system, but none of the users know Java well enough to feel confident writing interpreted Java scripts (and that's too close to JavaScript for my comfort). I once tried swapping this out for the Rhino JavaScript interpreter and got all the verbal support in the world but no developer time to design an API that'd work for all the departments.

The server isn't much better though. It's a tomcat based application that was written by someone who had never built a tomcat application before, or any web application for that matter and uses raw SQL strings instead of an orm, it doesn't use MVC in any way, and insane amount of functionality is dumped into the jsp files.

It too interacts with a raster image processor to create difference masks of the output, running PostScript as needed. It spawns off multiple threads and can spend days processing hundreds of gigabytes of image output (depending on the size of the tests).

We're stuck on Tomcat seven because we can't upgrade beyond Java 6, which brings a whole manner of security issues, but that eager little Java updated will break the tool chain if it gets its way.

Between these two components we have the Java RMI server (sometimes) working to help generate image data on the client side before all images are pulled across a UNC network path onto the server that processes test jobs (in PDF format), by reading into the xref table of said PDF, finding the embedded image data (for our server consumed test files are just flate encoded TIFF files wrapped around just enough PDF to make them valid) and uses a tool to create a difference mask of two images.

This tool is very error prone, it can't difference images of different sizes, colour spaces, orientations or pixel depths, but it's the best we have.

The tool is installed in both the client and server if the client can generate images it'll query from the server which ones it needs to and if it can't the server will use the tool itself.

Our shells have custom profiles for linking to a whole manner of third party tools and libraries, including a link to visual studio 2005 (more indirectly related build dependencies), the whole profile has to ensure that absolutely no operating system pollution gets into the shell, most of our apps are installed in our home directories and we have to ensure our paths are correct for every single application we add.

And... Fucking and!

Most of the tools are stored as source bundles in a version control system... Not got or mercurial, not perforce or svn, not even CVS... They use a custom built version control system that is built on top of RCS, it keeps a central database of locked files (using soft and hard locks along with write protecting the files in the file system) to ensure users can't get merge conflicts by preventing other users from writing to the files at all.

Branching is heavy weight and can take the best part of a day to create a new branch and populate the history.

Gathering the tools alone to build the Dev environment to build my project takes the best part of a week.

What should be a joy come hardware refresh year becomes a curse ("Well fuck, now I loose a week spending it setting up the Dev environment on ANOTHER machine").

Needless to say, I enjoy NOT working with Java. A lot of this isn't Javas fault, but there's a lot of things that Java (specifically the Java 6 version we're stuck on) does not make easy.

This is why I prefer to build my web apps in python or node, hell, I'd even take Lua... Just... Compiling web pages into executable Java classes, why? I mean I understand the implementation of how this happens, but why did my predecessor have to choose this? Why?

tl;dr i am proud of my universal program but annoyed it won't get appreciation.

<brag type='slightly'>the last three days i refactored my various snippets to a kind of modular and scalable software package. restricted to a rigid company system i make use of the technologies i feel confident in. so i created a javascript app that can be used with internet explorer. it is a neat tool to work smarter and mainly to make repetitive writing tasks efficient using predefined textblocks that have automated linguistic adjustments and are multilingual usable. after refactoring it is possible to extend any desired functionality by just adding another module. i learned a lot about implementing separated data structures, data processing, output and asynchronous script loading (and the annoying limitations of ie11).</brag>

i kept in mind that this tool might not only help my personal duties to be done more efficient but also might come in handy to all my colleagues having similar tasks to do. the downside is my colleagues having irrational computerphobia and i know for sure they will proceed to do these repetitive writings manually resulting in inconsistencies and an inefficient time management. while my wise wife tries to convice me that at least i had fun coding this stuff and having it supporting me with annoying tasks, it still bothers me being the only user, as it means no progression for the company. it riddles me how the colleagues, acknowledging us all being craftspeople in the first place, avoid use of computers whenever possible and rather rely on medieval working flows.

i find it quite amusing to be the 'can you fix my printer'-guy, but i just cannot handle this attitude. and everyone complains about having so much to do. get your shit together and start clicking these few buttons goddammit!

!rant (I got down voted for this on Stack Overflow, so I try to discuss the issue with a more professional crowd.)

In a Software Engineering class, we had an assignment to read Parnas' seminal paper on modularization [0]. In this paper, two approaches of dividing a software into modules are discussed:

Traditional Approach: A flow chart is drawn to work out the single processing steps and the program's high-level flow. Then every processing step is turned into a module. This approach doesn't yield very good results.
New Approach: Every design decision will be turned into a module by the means of information hiding. This approach leads to much better results.

My personal interpretation of the term design decision is that the modules are identified as data structures rather than as processing steps of an algorithm. This makes sense, because data structures are much more suitable for information hiding then processing steps of an algorithm. (The information inside a data structure is hidden behind functions, whereas a function only hides more detailed processing steps and no information; the information is actually passed in as arguments.)

Why does the second approach work so much better than the first approach? Here comes my second interpretation: The single processing steps of an algorithm are not replaceable (and thus not reusable), whereas it's possible to convert data structures into other data structures.

And here's my question: Could that be the reason why software development using workflow engines (based on BPMN, for example) never really took off?

My personal experience is that the activities created in such workflows are hardly ever reused, but there often are big data structures passed around all the involved activities, even if most of the activities use only one or two of them.

My question exaggerated: Could we get rid of all those clumsy workflow engines by giving managers Parnas' paper to read?

[0]: On the criteria to be used in decomposing systems into modules (Parnas 1972)

Can anyone help me with this theory about microprocessor, cpu and computers in general?

( I used to love programming when during school days when it was just basic searching/sorting and oop. Even in college , when it advanced to language details , compilers and data structures, i was fine. But subjects like coa and microprocessors, which kind of explains the working of hardware behind the brain that is a computer is so difficult to understand for me 😭😭😭)

How a computer works? All i knew was that when a bulb gets connected to a battery via wires, some metal inside it starts glowing and we see light. No magics involved till now.

Then came the von Neumann architecture which says a computer consists of 4 things : i/o devices, system bus ,memory and cpu. I/0 and memory interact with system bus, which is controlled by cpu . Thus cpu controls everything and that's how computer works.

Wait, what?
Let's take an easy example of calc. i pressed 1+2= on keyboard, it showed me '1+2=' and then '3'. How the hell that hapenned ?

Then some video told me this : every key in your keyboard is connected to a multiplexer which gives a special "code" to the processer regarding the key press.
The "control unit" of cpu commands the ram to store every character until '=' is pressed (which is a kind of interrupt telling the cpu to start processing) . RAM is simply a bunch of storage circuits (which can store some 1s) along with another bunch of circuits which can retrieve these data.

Up till now, the control unit knows that memory has (for eg):
Value 1 stored as 0001 at some address 34A
Value + stored as 11001101 at some address 34B
Value 2 stored as 0010 at some Address 23B

On recieving code for '=' press, the "control unit" commands the "alu" unit of cpu to fectch data from memory , understand it and calculate the result(i e the "fetch, decode and execute" cycle)

Alu fetches the "codes" from the memory, which translates to ADD 34A,23B i.e add the data stored at addresses 34a , 23b. The alu retrieves values present at given addresses, passes them through its adder circuit and puts the result at some new address 21H.

The control unit then fetches this result from new address and via, system busses, sends this new value to display's memory loaded at some memory port 4044.
The display picks it up and instantly shows it.

My problems:
1. Is this all correct? Does this only happens?

2. Please expand this more.
How is this system bus, alu, cpu , working?
What are the registers, accumulators , flip flops in the memory?
What are the machine cycles?
What are instructions cycles , opcodes, instruction codes ?
Where does assembly language comes in?
How does cpu manipulates memory?
This data bus , control bus, what are they?
I have come across so many weird words i dont understand dma, interrupts , memory mapped i/o devices, etc. Somebody please explain.

Ps : am learning about the fucking 8085 microprocessor in class and i can't even relate to basic computer architecture. I had flunked the coa paper which i now realise why, coz its so confusing. :'''(

Anyone tried converting speech waveforms to some type of image and then using those as training data for a stable diffusion model?

Hypothetically it should generate "ultrarealistic" waveforms for phonemes, for any given style of voice. The training labels are naturally the words or phonemes themselves, in text format (well, embedding vectors fwiw)

After that it's a matter of testing text-to-image, which should generate the relevant phonemes as images of waveforms (or your given visual representation, however you choose to pack it)

I would have tried this myself but I only have 3gb vram.

Even rudimentary voice generation that produces recognizable words from text input, would be interesting to see implemented and maybe a first for SD.

In other news:
Implementing SQL for an identity explorer. Basically the system generates sets of values for given known identities, and stores the formulas as strings, along with the values.
For any given value test set we can then cross reference to look up equivalent identities. And then we can test if these same identities hold for other test sets of actual variable values. If not, the identity string cam be removed, or gophered elsewhere in the database for further exploration and experimentation.

I'm hoping by doing this, I can somewhat automate the process of finding identities, instead of relying on logs and using the OS built-in text search for test value (which I can then look up in the files that show up, and cross reference the logged equations that produced those values), which I use to find new identities.

I was even considering processing the logs of equations and identities as some form of training data perhaps for a ML system that generates plausible new identities but that's a little outside my reach I think.

Finally, now that I know the new modular function converts semiprimes into numbers with larger factor trees, I'm thinking of writing a visual browser that maps the connections from factor tree to factor tree, making them expandable and collapsible, andallowong adjusting the formula and regenerating trees on the fly.

When you write an efficient piece of code and because the legacy database is too slow to return the data, you have to find a way to delay your processing instead of finding a way to speed up the database web service.

Automate this!

I'm an aspiring coder working some chappy administrator job just to pay the bills for now. My boss found out that I may actually be more computer literate than I let on.

Boss: "I want you to make X happen automatically if I click here on this spreadsheet"

Me "X!? That means processing data from 4 different spreadsheets that aren't consistently named and scraping comparison info from the fronted of the Web cms we're using"

Boss: "if you say so.. Can you do it?"

Me: "maybe.. Can I install python?"

Boss: "No..."

Me: "what about node.js or ruby?"

Boss: "no.. I don't know what you're talking about but you're not installing anything, just get it done"

Me: "Errm Ok.."

So here I am now, way over my head loving the fact that I'm unofficially a Dev and coding my first something in Powershell and vb that will be used in business :)

Sucks that I still have to keep my regular work on target whilst doing this though!

1) Had to fix severe bugs in a dynamic UI (configuration-driven forms) component.

Recognized undocumented Copy/Paste/Modify/FuckUp driven variations of the same component all over the project. Unsurprisingly, the implementations covered 99% of the antipattern catalog on wiki.c2.com and could compete with brainfuck in regard to human-readable code.

Escalated the issue, proposed a redesign using a new approach, got it approved.

Designed, Implemented, tested and verified the new shared and generic component. Integrated into the main product in the experimental branch. Presented to tech lead/management. Everyone was happy and my solution opened even more possibilities.

Now the WTF moment: the product with the updated dynamic UI solution never has been completely tested by a QA engineer despite my multiple requests and reminders.

It never got merged into baseline.
New initiatives to fix the dynamic UI issues have been made by other developers. Basically looking up my implementation. Removing parts they do not understand and wondering why the data validation does not work. And of course taking the credit.

2) back in 2013, boss wanted me to optimize batch processing performance in the product I developed. Profiling proved that the bottleneck ist not my code, but the "core" I had to use and which I must never ever touch. Reported back to him. He said he does not care and the processing has to get faster. And I must not touch the "core".

(FYI: the "core" was auto-generated from VB6 to VB.Net. Stored in SourceSafe. Unmaintainable, distributed about a bunch of 5000+ LoC files, eye-cancer inducing singlethreaded something, which had naive raw database queries causing the low performance.)

I need to make a legacy Java monstrosity asynchronous and came across a class which is currently making me cry blood.

It took our whole team 5 days to figure out how this thing works, including the weekend. At one place, It is adding an empty list, to another list.

The magic here is that if I remove the statement assigning data to the sub list, the data is still somehow being populated in the root list.

This clusterfuck somehow works in single threaded processing, but as soon as I make this multi threaded, all hell breaks loose.

Please send help!!

Harari said of the idea of Data-ism:

---

In its extreme form, proponents of the Dataist worldview perceive the entire universe as a flow of data, see organisms as little more than biochemical algorithms and believe that humanity’s cosmic vocation is to create an all-encompassing data-processing system — and then merge into it.

We are already becoming tiny chips inside a giant system that nobody really understands. Every day I absorb countless data bits through emails, phone calls and articles; process the data; and transmit back new bits through more emails, phone calls and articles. I don’t really know where I fit into the great scheme of things, and how my bits of data connect with the bits produced by billions of other humans and computers. I don’t have time to find out, because I am too busy answering emails.

---

I was initially entertained by the punchline, but that was soon followed by the rather depressing realisation that my only value to greater society is essentially as a data processing unit

Got a new job in data processing. Took them a couple days to get me a laptop and two weeks after that to get the core programs I need to begin working. Found out today that I *still* don't have access to everything I need for my job.

Hardware classes for software dev student?

Hey guys. Currently getting into second year of a 5 year curriculum to get an 'Integrated Master of Computer Engineering & Informatics' Degree here in Greece.

I'm already into software, I'm fooling around with java, go and php, making some games, web services and anything I find interesting in general. Recently, with the logic design class, I started liking hardware stuff (I didn't really like them before).

We're getting to a point where we might have to decide between picking hardware-centered or software centered subjects. I'm thinking that I can probably learn whatever is taught on the software side by myself (with a bit more studying of course), whereas hardware would be more difficult to study alone.

That said, I'm considering picking hardware, but I am skeptical. What do you think? I'll certainly miss out on the concurrent processing, data structure and how-a-compiler-works classes.

What do you think?

P.S. University here is free

I took a project. Wild mix of php and html including db stuff and data processing. About 200 files, some 3000+ lines long with if else cases processed in another template/logic behemoth...

I wrote a js file included it at the footer of the monster and update dom + data via ajax on my own api implementation because I'm too afraid to write in any of those files.

I've been told its quality code and well documented

Make your code available for your team members, please.

So we're working on this robotics project using ROS, a framework that enables multiple nodes in a network exchange their functionality among each other through tcp connections. Each node can be implemented and executed on your own machine, and tested with dummy inputs, but in collaboration they make a robot do fancy stuff.

The knowledgebase needs data from the image processing unit, providing this data to others with semantic context to high level planning, which uses this semantic data for decision making and calling the robot manipulation node with meaningful input, to navigate the robot's components in the environment. We use a dedicated machine, which pulls the corresponding repositories and is always kept configured correctly, to run each node, such that everybody has access to each other's work when needed.

So far so good. We tried to convince the manipulation guy (let's call him John) to run his code on our central machine, not a week, but since the first day, 5 months ago. Our cluster classification has been unavailable for 2 months, but my collegue fixed that. We still can't run the whole project without John's computer. If his machine blows up we're fucked.

Each milestone feels like a big-bang-test, fixing issues in interfaces last-minute. We see the whole demo just moments before our supervisors arrive at the door.

I just hope he doesn't get hit by a truck.

Today I learned that in our team, where we usually process data for runtime usage through batch scripts, which is the dumbest shit anyone can think of, someone decided to do data processing through VBA inside an excel file.

So that proves, regardless of how bad a solution is, an even more stupid solution is still possible.

At least it's not documented, so my hope is no one will see and copy it.

Girlfriend (a biologist): babe! I'm working on an R script to automate processing and analyzing out field data!
Me: 😳 *rethinking the idea of marriage*

Was working in an n-tier website, standard Web forms, BAL, DAL, database architecture. Validation and processing of data done in the BAL. Not the best idea, but whatever. Well apparently some developer thought it was too much work to pass his data through the BAL, so he directly accessed the DAL, performing zero validation on the data being passed in. Luckily, this was in a non-critical part of the site but the PM at the time nearly had a heart attack when I told him.

Does anyone really use Processing for Data Visualization? It's the fucking worst tool ever for visualizing data.
They barely have any forums either.

Spend 3 hours on a processing logic. Had it right the first time, thought the data was wrong.

Anyone out there a Scala fan? I am! Seems like most of the stories here are quite negative but positively is my thing so here's my 2 cents:

Scala is an amazing data processing language. It's a functional language with a lot of really great things like a consistent collection library api, case classes, brilliant async library's like Akka Actors, and plenty of solid learning resources like Twitter school and Martin Odersky's online course.

Oh... I dont know what to pick...
So i will pick 3 projects from my 3 stages of my dev "carrier"

1.Right after i discovered programing and learned how ,if, while and similar structures worked. The launguage was object pascal with delphi 2007
That was a "safe" with a stupidly complicated lock (text inputs, sliders, ect) it opened a secret folder in the end.

2. It was a embedded code for a Atmega8 AVR, Atmel studio, pure C but without memory managment (i didnt even know that it even existed)
It was a Pip boy knockoff, 16x2 display and a small keyboard connected to the arduino like board that i made on a proto board.
It wasnt that much of a pipboy, it was more of a showoff of atmega8 internal systems, (ADC, timers, interrupts and such)

3.DataLab, after helping my friend with his master thesis, (we meet on discord long story, i was in high school) i decided that mathlab is shit and i created a visual scripting enviroment, launguage C# .net 4 (in the latest version)
I remade the whole program from scrach 1 time, significantly improving everything (code reuse, better algorithms, data processing, code redability and edge cases) I have learned good practises from everywhere. I learned how to use git.

DataLab project looks just like LabViev (i didnt notice that it even existed...), it is frozen now because of my mental status but im planning on using it on my CV when i will be looking for jobs on holidays. There are many things that i can improve in that program but ... first i have to fix myself.

My university is offering Mobile Computing and Large Scale Data Processing as electives. Which one do you think would be more useful in the future?

I'm currently a java developer. I've dabbled in python too. Mostly worked on API development and some data processing. I want to learn something new, that'll keep me engaged. It can be something within java (like image processing or NLP) or some other language (Go, scala, js). What do you all suggest?

They increased 1 single file to have over 1.6 million records of data and bow the processing takes 12h to complete. They want me to improve the current bash scripts to decrease this processing time down to max 5 hours. Are you serious rn. Do i look like a magic fcking wizard 🧙🏿‍♂️🪄

OK, so I've been working on processing a Japanese dictionary file and things are going smoothly for the most part. Out of ~185,000 entries, I've got 35 that are still causing problems.

The error I'm getting is "Incorrect string value '\xF0\xA4\xAD\xAF' for column...". I've checked all of my encoding and collation settings, and I'm pretty sure I've got it set to properly implement all of Unicode (as well as it does, anyway), as shown in the image attached. My suspicion is the problem characters are likely among the JIS X 0213 character set; in either case we're clearly dealing with a 4-byte character encoding issue here.

If needed I can attach a flag in the database and base64 encode these particular entries so the data isn't lost, but I'd like to just get it to handle the data properly in the first place if possible.

Anyone have any ideas on other items I can check to resolve the error?

"No, the Client doesn't like stored procedures so we have done all our TL parts of the ETL using a bunch of views on top of views on top of views."

Wish I could have been here at the start so I could have pushed back, sigh.

Siiiggghhhh, yet the client is anal about performance and even consistency in SSIS packages.....siiiggghhhhhh but we dont have SHOWPLAN permissions or even sp_who2 access...siiiigggghhhhhh.

If i expanded one of the final views, would be like 1k lines. For the amount of data, we move, there shouldn't be any noticeable processing time but it can take anywhere from 10mins to an hour.

Today I started a project in which I must parse and extract some features from orders. Features can be product names, options, custom data and more and then do some validations/processing.

The (main) problem ? All I have is a String per order and of course most of the product/options have either change or been deleted.

I want to sudo rm -rf myself 😞

Use nodejs for data-processing

"I have a lot of experience with 'Programs'."

- A customer, referring to our flagship equipment automation, and data acquisition/processing software suite.

Persisterising derived values. Often a necessary evil for optimisation or privacy while conflicting with concerns such as auditing.

Password hashing is the common example of a case considered necessary to cover security concerns.

Also often a mistake to store derived values. Some times it can be annoying. Sometimes it can be data loss. Derived values often require careful maintenance otherwise the actual comments in your database for a page is 10 but the stored value for the page record is 9. This becomes very important when dealing with money where eventual consistency might not be enough.

Annoying is when given a and b then c = a + b only b and c are stored so you often have to run things backwards.

Given any processing pipeline such as A -> B -> C with A being original and C final then you technically only need C. This applies to anything.

However, not all steps stay or deflate. Sum of values is an example of deflate. Mapping values is an example of stay. Combining all possible value pairs is inflate, IE, N * N and tends to represent the true termination point for a pipeline as to what can be persisted.

I've quite often seen people exclude original. Some amount of lossy can be alright if it's genuine noise and one way if serving some purpose.

If A is O(N) and C reduces to O(1) then it can seem to make sense to store only C until someone also wants B -> D as well. Technically speaking A is all you ever need to persist to cater to all dependencies.

I've seen every kind of mess with processing chains. People persisting the inflations while still being lossy. Giant chains linear chains where instead items should rely on a common ancestor. Things being applied to only be unapplied. Yes ABCBDBEBCF etc then truncating A happens.

Extreme care needs to be taken with data and future proofing. Excess data you can remove. Missing code can be added. Data however once its gone its gone and your bug is forever.

This doesn't seem to enter the minds of many developers who don't reconcile their execution or processing graphs with entry points, exist points, edge direction, size, persistence, etc.

AWS offers a wide range of services that can be used to automate your IT operations. Some of the most popular services for automation include:

*AWS Systems Manager Automation: This
service allows you to automate tasks such as
provisioning servers, deploying applications, and
configuring security policies.
*AWS Lambda: This service allows you to run
code without provisioning or managing servers.
This can be used to automate tasks such as
sending emails, updating databases, and
processing data.
*AWS CloudFormation: This service allows you
to create and manage infrastructure as code.
This can be used to automate the deployment
of complex IT environments.
*AWS CodePipeline: This service allows you to
automate the software development lifecycle.
This can be used to automate the build, test,
and deploy of applications.

While the sidenote of explanation regarding the business process being added up front without much useful detail is all nice etc

Expecting some overblown explanation about fixing a mistake from a middle aged man as to motivation is just stupid

What else is stupid is directing a person to the same things when you people are supposedly supposed to spare us that

What else is stupid is people processing literally 10s of 1000s of man hours of the PRECISE same work over and over again letting themselves be psychologically programmed and handicapped

And what is really dumb is when vital data that can make a large difference in a payment getting processed or a claim being accepted or rejected is just allowed to pass through entirely on the premise that it allows a broken ass system to bite some in the ass and give a break to someone else instead of FIXING THE FUCKING SYSTEM

Does anyone know of any OTJ type places in the CSRA or online? Specifically in design or data processing? Online is great too.