Distributed systems be like

My Professor today, explaining data distribution in distributed systems:

"Imagine distributing Username data in subsets, such as A-C, D-F, G-I, J-L etc... And we have a lot of users with A .... *long pause* A bunch of assholes basically .."

Me: Boss, i am not qualified for this. This is something totally different that what i do.
Boss: Just do what you can.
* Me does something which seems to work*
-- A few months or even years later:
Boss: Our distributed systems don't longer work. What happened?
Me, after checking different system: Oh, there is a key that expired. I didn't know this key had an expire date. So they can no longer connect.

Turns out we have to visit every remote system (driving distance of a few 100's km) and set a new key. We couldn't do it remotely since we lost access.

Maybe, just maybe, when your employee says he isn't qualified for a task, listen and search someone that know what he is doing.

My colleague unknowingly uses the word "paralyze" for "parallelize". When is the good time to correctly him? So far he has successfully paralyzed my code.

*In a lecture

Lecturer: Consequences of Distributed Systems
Students: **attentive as fudge**
Lecturer: Heteroginity
Students:
Lecturer: Independent Failures
Students:
Lecturer: No global 'COCK'
Students: WHOOOOA, global what?...that escalated fast
Lecturer: I mean no global 'CLOCK'
Students: yeah right, can we use cock though?

To me, it seems like the rise of distributed systems like mesos / kubernetes combined with Docker require you to be master sysadmin, veteran kernel hacker and a part time c developer ALL AT ONCE if you really want to shave off time from debugging/ performance tuning sessions. Anyone wish they paid more attention in class ? Lol.

Debugging distributed systems in a nutshell.

Course title: Advanced Database Management

Course Objectives:
-Create a database with SQL.
-Describe data normalization of database information.
-Describe distributed database management system.
-Design databases based on Entity Relationship modeling.
-Discuss connecting to databases with server-side scripts.
-Discuss database administration and security.
-Discuss database systems

Like. Come. On.

Developer vs non developer interview:

Non developer:
How well do you know excel ?

Developer:
How would you write spreadsheet app, what if it was cross platforms mobile application but also desktop app ?

Non developer:
Do you know how to use windows?

Developer:
Do you know kubernetes, distributed systems, lambdas, cloud services and how to deploy to server farm ?

Non developer:
You know how to use printer / fax and coffee machine ?

Developer:
Do you have experience in writing code for embedded devices ?

Non developer:
Do you know powerpoint ?

Developer:
How well do you know javascript / html / css, are you comfortable with writing backend node.js code or electron applications ?
React native and native apps maybe ?

About slightly more than a year ago I started volunteering at the local general students committee. They desperately searched for someone playing the role of both political head of division as well as the system administrator, for around half a year before I took the job.

When I started the data center was mostly abandoned with most of the computational power and resources just laying around unused. They already ran some kvm-hosts with around 6 virtual machines, including a cloud service, internally used shared storage, a user directory and also 10 workstations and a WiFi-Network. Everything except one virtual machine ran on GNU/Linux-systems and was built on open source technology. The administration was done through shared passwords, bash-scripts and instructions in an extensive MediaWiki instance.

My introduction into this whole eco-system was basically this:
"Ever did something with linux before? Here you have the logins - have fun. Oh, and please don't break stuff. Thank you!"

Since I had only managed a small personal server before and learned stuff about networking, it-sec and administration only from courses in university I quickly shaped a small team eager to build great things which would bring in the knowledge necessary to create something awesome. We had a lot of fun diving into modern technologies, discussing the future of this infrastructure and simply try out and fail hard while implementing those ideas.

Today, a year and a half later, we look at around 40 virtual machines spiced with a lot of magic. We host several internal and external services like cloud, chat, ticket-system, websites, blog, notepad, DNS, DHCP, VPN, firewall, confluence, freifunk (free network mesh), ubuntu mirror etc. Everything is managed through a central puppet-configuration infrastructure. Changes in configuration are deployed in minutes across all servers. We utilize docker for application deployment and gitlab for code management. We provide incremental, distributed backups, a central database and a distributed network across the campus. We created a desktop workstation environment based on Ubuntu Server for deployment on bare-metal machines through the foreman project. Almost everything free and open source.

The whole system now is easily configurable, allows updating, maintenance and deployment of old and new services. We reached our main goal for this year which was the creation of a documented environment which is maintainable by one administrator.

Although we did this in our free-time without any payment it was a great year with a lot of experience which pays off now.

So today I'm going to join Reddit
Tell me subreddits where I can learn more and at the same time it doesn't go too much above my head

Interests in which I'd like to invest time:
Machine learning
Statistics
Distributed systems
Or general computer science research

Suggest me

In 2015 I sent an email to Google labs describing how pareidolia could be implemented algorithmically.
The basis is that a noise function put through a discriminator, could be used to train a generative function.
And now we have transformers.

I also told them if they looked back at the research they would very likely discover that dendrites were analog hubs, not just individual switches. Thats turned out to be true to.

I wrote to them in an email as far back as 2009 that attention was an under-researched topic. In 2017 someone finally got around to writing "attention is all you need."

I wrote that there were very likely basic correlates in the human brain for things like numbers, and simple concepts like color, shape, and basic relationships, that the brain used to bootstrap learning. We found out years later based on research, that this is the case.

I wrote almost a decade ago that personality systems were a means that genes could use to value-seek for efficient behaviors in unknowable environments, a form of adaption. We later found out that is probably true as well.

I came up with the "winning lottery ticket" hypothesis back in 2011, for why certain subgraphs of networks seemed to naturally learn faster than others. I didn't call it that though, it was just a question that arose because of all the "architecture thrashing" I saw in the research, why there were apparent large or marginal gains in slightly different architectures, when we had an explosion of different approaches. It seemed to me the most important difference between countless architectures, was initialization.

This thinking flowed naturally from some ideas about network sparsity (namely that it made no sense that networks should be fully connected, and we could probably train networks by intentionally dropping connections).

All the way back in 2007 I thought this was comparable to masking inputs in training, or a bottleneck architecture, though I didn't think to put an encoder and decoder back to back.

Nevertheless it goes to show, if you follow research real closely, how much low hanging fruit is actually out there to be discovered and worked on.

And to this day, google never fucking once got back to me.

I wonder if anyone ever actually read those emails...

Wait till they figure out "attention is all you need" isn't actually all you need.

p.s. something I read recently got me thinking. Decoders can also be viewed as resolving a manifold closer to an ideal form for some joint distribution. Think of it like your data as points on a balloon (the output of the bottleneck), and decoding as the process of expanding the balloon. In absolute terms, as the balloon expands, your points grow apart, but as long as the datapoints are not uniformly distributed, then *some* points will grow closer together *relatively* even as the surface expands and pushes points apart in the absolute.

In other words, for some symmetry, the encoder and bottleneck introduces an isotropy, and this step also happens to tease out anisotropy, information that was missed or produced by the encoder, which is distortions introduced by the architecture/approach, features of the data that got passed on through the bottleneck, or essentially hidden features.

My status:
Graduate student studying Computer Science at the University of New York at Buffalo.
4 Computer Vision and Image Processing Projects
3 Distributed Systems projects (Android apps).
Red Hat Certifications.
Applied to 135 companies for an internship program.
Here are the replies I have gotten so far.
" We have analyzed your resume and we think you'll be a great choice for this position at our company."
What position?
MARKETING INTERN.
FML!

Debugging your application on a mesosphere cluster for the first time.

Everyone I tell this to, thinks it’s cutting edge, but I see it as a stitched together mess. Regardless:

A micro-service based application that stages machine learning tasks, and is meant to be deployed on 4+ machines. Running with two message queues at its heart and several workers, each worker configured to run optimally for either heavy cpu or gpu tasks.

The technology stack includes rabbitmq, Redis, Postgres, tensorflow, torch and the services are written in nodejs, lua and python. All packaged as a Kubernetes application.

Worked on this for 9 months now. I was the only constant on the project, and the architecture design has been basically re-engineered by myself. Since the last guy underestimated the ask.

Hey DevRant!

Not really a rant but a question:
I just got accepted into a coding bootcamp. Have any of you been involved in one? How was it? What would you do to make it a better experience throughout? Any advice or suggestions?

It's full time, six months long and I start in October and I want to make sure I make the most of the experience and absorb as much as possible.

I'm super happy that the course appears to be less just learning JavaScript and more involved in the Computer Science side of things, even including bits about C/C++, distributed systems, algorithms and data structures, software design/testing, cryptography, database management, and computer architecture. It also, of course, covers tons of resume work, interview practice, and networking.

Thanks!

Scheduled an on-site. *internal screaming*

Does anyone have any resources for studying distributed computing and operating system topics or have any pointers for studying for a systems design interview?

Also, how did y’all get comfortable with recursion? I don’t have issues with problems I already know the solutions to but it’s like when that’s not the case my brain just goes into panic mode for a bit.

Teach me your ways?

Whoever the fuck at my university thought that a distributed systems project using Java Web services was a good idea? The server we're supposed to use (Glassfish) is so out-of-date, half the time spent on this project is just spent fixing fucking broken dependencies and otherwise getting it to play nice. Please just tell me this shit isn't used in industry outside of legacy applications.

A new development rule I've started to implement:

All backend APIs will be written with the assumption that it's gonna get distributed as an API for 3rd parties to be integrated in their systems - meaning that every API I write will have proper response status codes for appropriate scenarios (like 400, 429, 500 status codes).

No more `res.json({status: false, message: 'message'})` with 200 status code across the board.

So do these (+/- 10ms) add or are they a universal “+/- 10ms from the actual time on stratum 0?”

I missed this last week... so too bad ;)

My introduction into programming was rather slow. When I was a child, we had an Apple IIc, but there were no disks. When you'd boot it up, you got a prompt and I recall being able to type commands into it that someone told me was "Apple BASIC".

At the same time, our family computer was a 386 and it came with something called GWBasic. I was a huge Mortal Kombat fan as well, and I recall finding the moves for the game on an AOL usenet. I took them all and wrote a program in BASIC that let you search and find moves for your character. I distributed this on some floppies to friends.

After that I lost interest. My "Information Systems" shop in high school was more about how to use Office than it was about programming. A few years later I found out that you could run your own text-based games (MUDs) and I quickly jumped into that and the C language.

From there, I was in and out of programming - C, to C++. Java and PHP, then back to Java. It would be about 15 years later until I finally realized I wasn't bad at this and land a job doing it. :)

My final year taking a B.Sc. I'm writing up my Distributed Systems project, the day before handing it in. It's on top of Transis, and source code is "stored" in RCS (yes, I'm that old). The project is a reliable system administration tool, that performs the same action across a cluster with guaranteed semantics.
I'm very proud of the semantics, but cannot figure out why the subdirectory installation stuff works almost but not quite. Here's my sequence of actions:
1. Install across all machines.
2. Manually see it's broken.
3. "rm -rf *".
4. Repeat.

What in to discover is that the subdirectory installation always finishes off in a current directory 1 level higher than where it started. Oh, and the entire cluster sees my NFS home directory. Oh, and I'm running each cluster member in a deep subdirectory of my dev directory. Oh, and my RCS files live in a subdirectory of my dev directory.

All of a sudden, my 5 concurrent "rm -rf *"s were printing weird error messages about ENOENT and not being able to find some inodes. In a belated flash of brilliance, I figure out all the above, and also that I've just deleted my dev directory. 5 times, concurrently. And the RCS files.

That was the day a kindly sysadmin taught me than NetApps have these .snapshot directories.

Fuck companies that want you to build new systems that are a superset of what they already have when they can't even tell you what they have now. Fuck you with your bullshit, nonsensical, self-contradicting, third-assed diagrams--some half-UML, half-clip-art, half-pie-chart drivel. If you're the CTO of a company and you want me to think long and hard about rebuilding your distributed systems, you can think long and hard about expressing what you already have. And NO: a verbal explanation pow-pow WebEx over your DSL connection with your protege that mumbles worse than an Atlantan rapper doesn't count as fucking "knowledge transfer" of your fucking architecture, FOR FUCKS SAKE.

Currently on a project to use AI as music recommending model, an interactive AI model generator and researching on Distributed Systems altogether in three different teams for each thing...

Seems I'm the dirty fish of their team.. 😖

Fucking wasting the time all along the day...

Like Why the Fuck am I degrading my own potential, quality and what not...

Oh god Fucking pick me to the hell NOWWW

The first one was changing the main topic of my research activity, from distributed systems to AI/Deep Learning... Then moving from academia to companies, two very different worlds and two very different approaches, but I still like to think that my formal background is driving my decisions and helping me, even with everyday problems

Backend because I'm obsessed with distributed systems!

What is the best laptop for development?
Especially if you work a lot with virtualization, containers, the cloud, and a lot of systems programming. And you like to simulate a lot of distributed environments on locally?

Yo been a longtime.

So I basically quit my last job to have successfully reached the top company in my country only to find they are such a mess.

No code quality whatsoever, testing? Yiu crazy? And all the old people who think they are senior whilst they do not know jack..

I do distribured web applications, but shit I hate titles and I think of myself as a software guy, I can do software that opens the fridge when I close the toilet lid ffs!

So, I am looking to deviate my career from web to something more deep such as distributed systems and services where I can use all of my skills and expand my knowledge more, and be able to code in js, c++ golang and more, handle and tackle infrastructure issues, virtualization etc...

So I want to ask you guys what would be an interesting project I can work on to concretize my skill and be able to convince my next recruiter that I walk the talk.

Thank you everyone

My exams in Web Engineering and Distributed Systems at the University were postponed to mid April... Thank God, they would have started next Tuesday and I haven't properly learned. So I got that going for me.

I'm currently mostly learning web dev, but do you guys have any book recommendations for other fields of programming. Perhaps something low-level or something like distributed systems. Everything goes.

Been doing parallel programming and I’ll be taking a distributed systems course next semester. I’ve also been dabbling with Rasp Pis and have been enjoying working in linux/CLI and I’m considering getting building a cluster.

What are some use cases where I could put into practice distributed systems/parallel programming with cluster setups? No limits here :)

Git, Mercurial and others are distributed version control systems. Maybe it would have a federated version control system for hosting open source software...

I'm making a distributed system for my exam project, but the client have a weird idea when it comes to the webpage, that we havent learned about...
If customer A (my clients customer) opens client.com/Customer, an API should be used for customer A's DB data retrieval
If customer B uses the same site customer B's API should be used instead...
Any good way to differentiate API update a single API connection by the caller of the website?

Sticking with emacs as my favorite editor. Navigation within files is easy. Working on multiple files also. I don't have to leave my editor to use the shell and can manage my filesystem as well. And the most important feature for me is tramp. When working with distributed systems it is pretty nice to access the remote filesystems from your local machine.

The fact that Spark web ui let you go beyond year 200'000 ad on the job planning view is quite concerning
:/

Why is Java goto language for most backend systems, distributed and scalable, specially in big organizations like Amazon?

Leslie Lamport!