Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "scalability"
So last weekend I started collecting hardware for a small scale cluster at home to test scalability of my software. Making some decent progress.
Tomorrow I will replace the switch and this weekend I will set up storage so I can start my first application20
--- NVIDIA announces PhysX SDK 4.0, open-sources 3.4 under modified BSD license ---
NVIDIA has announced a new version, 4.0, of PhysX, their physics simulation engine.
Its new features include:
- A "Temporal Gauss-Seidel Solver (TGS)", an algorithm used in this SDK to make things such as robots, character arms, etc. more robust to move around. NVIDIA demonstrates this in the video by making their old version of PhysX, 3.4, seem like an unpredictable mess, the robot demonstrating that version smashing a game of chess.
- New filtering rules for supposedly easier scalability in scenes containing lots of both moving and static objects.
- Faster queries in scenes with actors that have a lot of shapes attached to them, improving performance.
- PhysX can now be more easily used with Cmake-based projects.
In essence, better control over scenes and actors as well as performance improvements are what's new.
Furthermore, NVIDIA has released PhysX version 3.4 under the 3-Clause-BSD-license, except for game console platforms.
As NVIDIA will release the new version on December 20th, it will also be released under the same modified BSD license as PhysX 3.4 is now.
What are your thoughts on NVIDIA making a big move towards the open-source community by releasing PhysX under the BSD license? Feel free to let us know in the comments!
A Developer is desperate: his java application servers are unresponsive, thousand of dead zombie threads are sucking all cpus, memory is leaking everywhere, garbage collector has gone crazy, the cluster sessions are fucked....
The Developer goes to the closest bridge, ties a stone to his neck and gets ready to jump.
Suddenly a bearded old man with a fiery look runs toward him, yelling:
- stop stop!!!! Your application is not scaling and misconfigured, your servers are melting, cpu usage is not sustainable anymore, but don't despair
The Developer, puzzled, looks at him:
-I've never seen you...how do you know...
- Hey, man, I'm the Devil. I know everything. All your problems are solved. I'll give you magic functions. They are called Lambda.
You'll never have to worry about your servers, scalability, security, configuration and shit.
The Developer seems astonished but relieved:
- Ok, sounds great! let's try it - suddenly suspicion creeps in - hmmmm but you are the Devil....so...you want something back, don't you?
(the Devil nods lightly with a diabolic smile)
- ...and...you want my soul, I guess...
- your soul??? come on!!! - the Devil burst in a laugh - we are in 2019. I don't care about your soul. I want your ass.
- yes, I want to fuck your ass
The Developer, evaluates quickly the situation.
Few moments of pain or slight discomfort (?) in exchange for magic lambda. It could be worth. He accepts.
After a while of rough anal fucking, the devil asks
- Hey, how old are you anyway?
- 45, why?
- Oh jeeez...45!!!??? and you still believe in the devil?5
When you about to start a project by doing a small module, but as you code, you keep thinking about scalability, security dependencies and such....
....three hours later, you still haven't written code2
Software engineering project discussion:
Boy: Sir, my project is a client to manage files stored on different cloud file storage systems at one place
Faculty: Boring idea. Very easy to implement, No scope of scalability, etc
Girl: My project is an app to display the weather information
Faculty: Omg! What an innovative idea! I'm surprised how no one though of this before!11
In my previous rant about IPv6 (https://devrant.com/rants/2184688 if you're interested) I got a lot of very valuable insights in the comments and I figured that I might as well summarize what I've learned from them.
So, there's 128 bits of IP space to go around in IPv6, where 64 bits are assigned to the internet, and 64 bits to the private network of end users. Private as in, behind a router of some kind, equivalent to the bogon address spaces in IPv4. Which is nice, it ensures that everyone has the same address space to play with.. but it should've been (in my opinion) differently assigned. The internet is orders of magnitude larger than private networks. Most SOHO networks only have a handful of devices in them that need addressing. The internet on the other hand has, well, billions of devices in it. As mentioned before I doubt that this total number will be more than a multiple of the total world population. Not many people or companies use more than a few public IP addresses (again, what's inside the SOHO networks is separate from that). Consider this the equivalent of the amount of public IP's you currently control. In my case that would be 4, one for my home network and 3 for the internet-facing servers I own.
There's various ways in which overall network complexity is reduced in IPv6. This includes IPSec which is now part of the protocol suite and thus no longer an extension. Standardizing this is a good thing, and honestly I'm surprised that this wasn't the case before.
Many people seem to oppose the way IPv6 is presented, hexadecimal is not something many people use every day. Personally I've grown quite fond of the decimal representation of IPv4. Then again, there is a binary conversion involved in classless IPv4. Hexadecimal makes this conversion easier.
There seems to be opposition to memorizing IPv6 addresses, for which DNS can be used. I agree, I use this for my IPv4 network already. Makes life easier when you can just address devices by a domain name. For any developers out there with no experience with administration that think that this is bullshit - imagine having to remember the IP address of Facebook, Google, Stack Overflow and every other website you visit. Add to the list however many devices you want to be present in the imaginary network. For me right now that's between 20 and 30 hosts, and gradually increasing. Scalability can be a bitch.
Any other things.. Oh yeah. The average amount of devices in a SOHO network is not quite 1 anymore - there are currently about half a dozen devices in a home network that need to be addressed. This number increases as more devices become smart devices. That said of course, it's nowhere close to needing 64 bits and will likely never need it. Again, for any devs that think that this is bullshit - prove me wrong. I happen to know in one particular instance that they have centralized all their resources into a single PC. This seems to be common with developers and I think it's normal. But it also reduces the chances to see what networks with many devices in it are like. Again, scalability can be a bitch.
Thanks a lot everyone for your comments on the matter, I've learned a lot and really appreciate it. Do check out the previous rant and particularly the comments on it if you're interested. See ya!25
Solved a complex puzzle on a website for a local ecommerce business, mind you in 16 and not really looking for a job but an unpaid internship would look beautiful on a resume or university application.
They wanted to see some of my code and give me a tour and none of them despite them being PHP developers for Magento could wrap their heads around laravel or how the routing worked. They also didn't understand and raw PHP whatsoever. I lost all faith and walked out of their office when they asked why I was using prepared statements and how they worked. That was after finding out that they don't understand cloud scalability whatsoever or common security practices.4
Management: List your most significant achievement during this cycle.
Me: Developed and implemented a full stack of database micro containers that allow scalability and reusability.
Yay, Dubba didnt die.
Okay, hello'ed back, lets fucking rant.
Im fucking SICK and TIRED and ANNOYED by all freaking cloud fanboys. Yeah, sure. You get scalability. You waste less resources. Sure. But it depends on use case. There is suprising ammount of services that can be run on VPS or dedicated servers that will be a lot cheaper if you just DONT run it on cloud. And dont you dare to missinterpret me. I fully recognize that there are valid uses for cloud services. But for fuck sakes, its not anwser for ALL of your needs, and it costs insane ammount of money if you use it where you should use dedicated.
WHY we, as humanity came to this place where self-claimed "proffesionals" will anwser you "cloud" before you even ask your question.
No, fuck off, take calculator and think for around 60 seconds. And than decide if you need cloud or not.
And no, this mail server will be fucking few times more expensive on cloud, and I know it becouse I have fucking calculator and I was running it past 1.5 year and actually run sometimes htop, so stop bullshitting me.
Sorry if some cloud die-hard felt offended but it is exacly how I see and feel it.
I got notified that tomorrow I'm gonna start a porting project from a FileNet ecosystem.
Well, I don't know what is FileNet, but at least I've enough time to study its architecture. Let's start from the official IBM page:
The FileNet® P8 platform offers enterprise-level scalability and flexibility to handle the most demanding content challenges, the most complex business processes, and integration to all your existing systems. FileNet P8 is a reliable, scalable, and highly available enterprise platform that enables you to capture, store, manage, secure, and process information to increase operational efficiency and lower total cost of ownership.
Thank you IBM, now I surely know how to use FileNet. Well, I hope that wikipedia explains me what it is:
FileNet is a company acquired by IBM, developed software to help enterprises manage their content and business processes.
Oh my god. I tried searching half an hour so far and everything I found was just advertisements and not a clue about what it is.
Then they wonder why I hate IBM so much5
My image of dream career through different times of my life:
- frontend specs prodigy, css enlightenment, a member of w3c or a similar committee
- indie hacker and entrepreneur, leader of a startup community
- architecture prodigy, expert in scalability
- transsexual evangelist, popular article writer and a rockstar
- hardware engineer: Linux, C, chip and dale’s Gadget-like girlfriends, xkcd, latex, assembly, buying a radio station and a telescope
- scientist like NickyBones, papers, data, more data
- art expert
Though achieving one of this would take the entire life, I had a chance to grasp all of this. WHY does they feel so incompatible? Why do I have to choose?
Why do I feel so sad? Why do I feel like I haven’t achieved anything even though I objectively achieved what I dreamed of like five years ago?
Is it true that it’s in my nature to always seek an environment to feel like a junior in? Is feeling like a junior only pleasant to me because it reminds me of old times when I wasn’t actually this mentally ill and was still happy?
Why do I feel like that arduino and C shit is the equivalent of a red corvette?6
I LOVE NODEJS
I LOVE ANGULAR
I LOVE WRITING MY OWN FRAMEWORKS
I LOVE ELECTRON
I LOVE IONIC
I DONT CARE OF APPS SCALABILITY
I HATE HYPSTERS !!!7
I spent a month trying to understand how to split our application and setup a pdf server to manage 10'000> pdf documents generation at the same time.
Today during our monthly meeting, our dear Key Account Manager, said he's not selling pdf generation to customers since one year at least, and he's not planning to sell it in the future.
Nobody knew about that apparently. We had also some demo guys showing actually the functionality to potential customers.
Also nobody believe me when I say we should start a project to monitor used functionalities in our application.
Just wasted one month of my fucking extra-time. Now I'm late with a lot of other tasks, apparently more urgent that the potential scalability issue of a functionality we don't use.
Key Account Manager rules.
Key Asshole Manager.
I think we should hire a Key fucking-COMMUNICATION Manager
OMG. Just had the worst convo with my supervisor. She's a dev and she does not understand why scalability is important.
We have a really bad client who has a fucking insane amount of pages and they want to change some stuff on a few and my supervisor just said manually add it in with if satements.
I replied saying this client will want more things changed let me take the extra hlaf hour to make it scalable.
She responded no just do manually.
I wanted to get in to rant about how scalability is important but it was the end of the day and I was to tired.4
Finished a complete version upgrade with vastly improved scalability and maintenance in just 3 weeks!
I forgot to do the database migrations as I went along. Ok... no problem.. I'll just use some kind of database comparing software, I'm sure they exist...
...fuck... is there not any for free?
Ok fine whatever i'll manually compare the tables and columns, fuck it. Sure there's like 50 tables and like 20 columns per... whatever...
should take like a day or so, with testing...
Finished my new version upgrade after just 6 weeks of coding!
Salesforce. Although I wasn't involved in the purchase or the implementation, I spent many 100 hour weeks dealing with the crapshoot of an implementation. A large company abused that software to the point of no return. They used that thing for everything, and then they didn't even use it right for the one thing salesforce is good at. So I guess I don't have anything against salesforce itself besides its scalability issues, custom SOQL syntax, user model, and pricing. I'm more upset about the salesforce developers/business owners that decided it was okay to use salesforce for things it was never meant for, like inventory, project management, 3rd party sales team, and so many other things that caused what should have been sub-second queries to take 30 to 60 seconds.
Project manager: "What is a micro service? I'm dating a girl and she mentioned it and I want to impress her."
Me: "Well, you have monolithic services which tend to serve many different functions whereas a micro service tends to serve a single function or a few related functions. They are usually easier to scale and can be optimized to be faster. Still, right tool for the job."
Project manager: "Oh nice! So I can ask her 'Hey, want to see my micro service? It's quick and scalable.'"
He's leaving this week. I'm going to miss him though.
Seriously though, in that context, would scalability mean you're bringing friends?4
So I did this https://devrant.io/rants/797965/... which works fine until medium sized data.
However for large data the ETL pegs a 6 core Xeon (2.2GHz) with 50GB of ram. Because of it ends up doing six threaded compares, so 12 different data sets. Other than "pull less data", any tips?
Code (C#) is basically a Linq multi column join between two DataTables and when the compared columns don't match it returns as a var which is turned into a third DataTable to be SqlBulk loaded into the DB.
Table1 is external API return data (no windowing) and Table2 is from our DW.7
Whether it's a good idea to host my very primitive web-app on Heroku or setting up my own VPS on DigitalOcean etc.?
Heroku will save me some time, but I'm not sure about the scalability and cost.
VPS approach means I'll have to take care of all the DevOps myself, even though I know a bit Linux to kick things off, this is still quite daunting to me.
Any suggestions? Insights? Rants?12
I prefer it doing 2 tasks parallely during the initial phase of requirement gathering and design phase.(makes more sense if you are working extremely new system and framework)
1. Keep collecting requirements from clients and understand them.
2. Collect different designing aspects for the project and parallely, build a POC for 2 purpose: to get hands into the new Framework and also as a demo to clients. Working on POC helps in 3 ways: Improving understanding of requirement, improving framework knowledge, and playing around with code whenever bored of designing and reading tons of existing designs..
3. Once primary requirements are clear and fixed, analyse all different designs, if possible I setup meetings with senior devs, principal engineers (they help a lot when it comes to reviews on scalability and reliability of a design)
4. The above design is mostly architectural level. Once design is fixed, then I start taking each component and prepare a detailed implementation design. (Notice that whenever I am bored of designing, I spend sometime in building POC)
5. In detail design, I focus on modularity and flexibility. Anything defined should have getters and setters for example. This will help you reuse your code. Keep the interface between components in your design as generic as possible, so that in case your MySQL is change to Postgre or NoSQL, your design should be able to adapt new features..
6. Instead of building entire project, define feature targets and deliver small features.. this will help you to be in line with the requirements with minimum deviation.
My 2 cents on different OSes to use.
I think Linux is best for running servers and services and having long run times with little issues (when its Console and not GUI based.) But I have a lot of issues with using its GUI distributions like Ubuntu and it feels kind of unpolished in that area.
I prefer macOS for its GUI as it actually works and has far less issues than Windows GUI and is (IMO) better than Linux GUI's by far. But macOS just doesn't feel like it was designed super users and it can feel like its holding you back a bit. Also you have to use Mac hardware which are amazing machines, they are just overpriced.
I prefer Windows for its GUI and despite its problems, it is very well designed for super users and has very well designed remote desktop features and scalability (although it is a pain to maintain.) Windows works well for connected company systems.
In my opion:
Linux: Servers, databases (no GUI)
macOS: Designers, photo/video editing, IT/programmers and general use as a standalone (not part of a company system).
Windows: IT/programmers, super users, general use but better than macOS at working together in a company setup, but macOS is better at being a personal laptop or PC.
I personally use Linux for our email and web servers. Windows for our company computers (designers use Macs) and I have a Macbook as my own personal computer.33
I’m working on a react codebase and company decided to add a new module.
Now im writing markup and css to ensure UX is smooth as designers thought of it.
Imagine my horror when I start to code and find out no matter what HTML tag i use, it’s been FUCKING OVERRIDDEN in the global stylesheet. AND STYLES HAVE BEEN OVERRIDDEN WITH !important
They’re also using Ant design as a component library. Guess what, default ant design classes have been overridden too. So i try to use ant design button or card, and bam, MAGICALLY SOME DESIGN FROM SOME SHITHOLE MODULE DECIDES TO FUCK WITH MY STYLES
On top of that, styles of parts of application has been written in SASS, some part of application uses bootstrap components some use third party components like tables and responsive grids to suit to their preferences. Some parts use handwritten css. Some parts use CSS IN JS and styled components. THE FUCK IS THIS GARBAGE!!!! THE FUCKING CODEBASE HAS A MIND OF ITS OWN!!!!!! YOU NAME A WAY TO ADD STYLES TO A COMPONENT, ITS THERE!!!
And the company’s management thought a “fractal” approach to maintain each individual view is “best” for SCALABILITY!!! HOW THE FUCK DID IT NOT CROSS YOUR DUMB MIND THAT FRACTAL APPROACH ALSO GUIDES TO HAVE ALL COMMON STUFF AT ONE PLACE!!!! THIS CODEBASE HAS DUPLICATE STYLES AND DUPLICATE CODE IN ALMOST EVERY MODULE!!!!
Not to mention every developer choosing to freely decide the way they should write their code without any guidelines.
HOW THE FUCK PEOPLE WRITE THEIR CODE WITHOUT THINKING ABOUT OTHER DEVS!!! SO BASICALLY I AM NOT ONLY CLEANING SOMEONE ELSE’S SHIT BUT ALSO TRY NOT TO SHIT IN THE PROCESS!! FML2
Me: “I think I’ll check linkedin today”
Clicks on video: “... good perspective on growth and blockchain micro service apis that leverage ML models for understanding interfaces that welcome scalability within a agile environment...”
Me: *jumps out window*2
University student here. I’m trying to learn how to build scalable applications. I figured what better way to learn, than to ask this community!
So, some questions:
1. What is a good stack for scalability
2. Regarding microservices, when and where should you use node, .net, and go
3. I hear much talk about docker and kubernetes... how would you use these concerning microservices
Half of the courses we had in our college were about electronics. Except Microprocessors and Transistors, it's not relevant.
We even had chemistry and engineering drawing. So we essentially wasted more than half of our time.
Besides languages, weren't taught anything about real world software development.
Nothing about how to work with an existing code base, version control, design patterns, system design, creating a website, debugging, functional programming, scalability, reliability.
The industry should be involved in setting the syllabus and also contributing part time teachers.2
showing it to my boss: "cool what you can do with html". WAT?
(he is not a programmer but knows i´m into web-programming)
I guess you could say that my speciality is cloud at scale. I’d say it chose me more than I chose it.
Looking back on it though, I think what I like about my speciality is the unique challenges it brings.
Every speciality has its own set of challenges, like tight resource limits in embedded, or client-server synchronisation in native/mobile.
The challenge of cloud at scale is throughput. Designing systems that can support 100K users making a bazillion requests a second, or a data pipeline firing events that you need to process in near real time without dropping a single one.
The real challenge of course is doing all this within a sensible budget. We have virtually infinite compute but we dont have infinite dollars to spend on it.
Its a fun problem to solve.3
So, the main system we develop and "maintain" is a horrible monolithic huge project that has the majority of the code in one and only one god class.
On top of that, all the main logic is implemented directly in the "main" method (yes, the "public static void main"). Now every time you have to create a method to use it in the main logic, it has to be static!
God, I hate to even look at this class, let alone touch it, I feel dirty, like I'm touching a pile of shit.
Have any of you fin devs out there on the interwebs had any experience with building social networks using `Stream`??
I am looking into their API for use in a golang project of mine but I am unsure about a few things...
2. Scalability and the cost of it
3. General API quality, in Golang specifically but any comments are great.
It's a stretch I'm sure but if there was a place that folks would know it would be here @ devRant1
Just migrated my custom help desk software for work to cockroachdb! Scalability here I come fucking come! And no more fucking MySQL Master Master configs to deal with.
I'll remove some courses - make some optional and some courses mandatory.
I'll explain- I did my B.E. degree in I.T. I'll remove some courses like the ECE subjects (Digital electronics , Communication Theory) - something I'll never use. If I will - I can learn it at that time. Some mandatory courses like DBMS, OS etc. And some optional ones you can take according to your passion like - security courses or scalability courses etc.
In Postgres, parallel query works by having multiple workers....
As such, depending on workers available, queries can run theoretically fast (workers available) or slow (workers not available) - right?
Question is: are there more optimizations like this... And is my assumption correct?
It seems like a major pain in the ass... Scalability and reliability wise. (?!)8
Guys check out IOTA, get a light node wallet and buy some as long as it is still low. It is rising at a rising rate since some hours now.
IOTA is the solution to bitcoins speed and scalability problems. IOTA does not use the blockchain but instead a network called the tangle, which enables decentralized peer-to-peer transactions. There are no miners, no fees and the transaction validation speed depends on how many people are using the currency. One transaction requires your device to validate 2 other transactions through proof of work and therefore the system can never be overloaded by too many transaction requests.
Be warned though, the IOTA foundation is only currently building up the infrastructure, and people are just starting to trade since a few weeks so trasactions still may take a few hours.3
Best book/source for learning everything devops'y'/kubernetes?
(Given that I have some sort of experience in Dockers, hosting websites and know a fracture of aws archeticture, but lack in good "cloud" thinking skills, scalability, understanding costs for production applications, cluster size, etc...)
Neo4j vs Mongo vs SQL
Which one would you choose for a high density scalable database?
Please comment with your ideas about speed and scalability.15
Not sure if I should implement Bootstrap on my website for cross device scalability, grid system placement and basic animations, or just stick to raw CSS3... Any suggestions?5
TIL stack exchange is built on ASP.NET
And there I thought the platform had serious scalability issues1
Struggling to optimize and to scale the infrastructure of our production environement dealing with people who don't bother themselves to write scalable code.
Hello fellow devs. I wanna create a survey site, which will contain 3 surveys within. All 3 surveys, will contain around 50 questions, and will have a range of input options. The site will collect the data from a form, store it and probably live visualize them with a plot-like library .
I was thinking about using React for the front-end, Firebase for the back-end + Google-Sheets and one of the many react-libraries for plotting.
Do you think that this stack is appropriate for that kind of app ? I am a bit concerned about firebase's querying system or the scalability of the app. (Since it will hold a lot of questions and answers )
( It's survey for developers, so I hope in the future you will participate in it :D ).
Any Erlang guys over here ?
I am getting into it after trying elixir, just to get solid foundations (understand it better and otp)
The goal is to get erlang elixir and pheonix undercontrol
Now my questions:
Any good reads about maintenance scalability in a prod environment ?
How to get good at solving problems.
My managers says that I should consider scalability, dev efforts, Operational costs etc. I am really new in this, I really need to figure out a lot of things by understanding architecture and then there are added quality parameters of the solution, where can I read all this? How to get good at this? I know one of the solution is to actually work on it, can I still get some other resources to understand things better?1