Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "relational data"
As a developer, sometimes you hammer away on some useless solo side project for a few weeks. Maybe a small game, a web interface for your home-built storage server, or an app to turn your living room lights on an off.
I often see these posts and graphs here about motivation, about a desire to conceive perfection. You want to create a self-hosted Spotify clone "but better", or you set out to make the best todo app for iOS ever written.
These rants and memes often highlight how you start with this incredible drive, how your code is perfectly clean when you begin. Then it all oscillates between states of panic and surprise, sweat, tears and euphoria, an end in a disillusioned stare at the tangled mess you created, to gather dust forever in some private repository.
Writing a physics engine from scratch was harder than you expected. You needed a lot of ugly code to get your admin panel working in Safari. Some other shiny idea came along, and you decided to bite, even though you feel a burning guilt about the ever growing pile of unfinished failures.
All I want to say is:
No time was lost.
This is how senior developers are born. You strengthen your brain, the calluses on your mind provide you with perseverance to solve problems. Even if (no, *especially* if) you gave up on your project.
Eventually, giving up is good, it's a sign of wisdom an flexibility to focus on the broader domain again.
One of the things I love about failures is how varied they tend to be, how they force you to start seeing overarching patterns.
You don't notice the things you take back from your failures, they slip back sticking to you, undetected.
You get intuitions for strengths and weaknesses in patterns. Whenever you're matching two sparse ordered indexed lists, there's this corner of your brain lighting up on how to do it efficiently. You realize it's not the ORMs which suck, it's the fundamental object-relational impedance mismatch existing in all languages which causes problems, and you feel your fingers tingling whenever you encounter its effects in the future, ready to dive in ever so slightly deeper.
You notice you can suddenly solve completely abstract data problems using the pathfinding logic from your failed game. You realize you can use vector calculations from your physics engine to compare similarities in psychological behavior. You never understood trigonometry in high school, but while building a a deficient robotic Arduino abomination it suddenly started making sense.
You're building intuitions, continuously. These intuitions are grooves which become deeper each time you encounter fundamental patterns. The more variation in environments and topics you expose yourself to, the more permanent these associations become.
Failure is inconsequential, failure even deserves respect, failure builds intuition about patterns. Every single epiphany about similarity in patterns is an incredible victory.
Please, for the love of code...
Start and fail as many projects as you can.30
New to this but here's my rant I suppose. It bothers me how "non-tech" people kind of devalue what tech people do. Like they have zero understanding of it, so you make something in 30 minutes or an hour that took years of building said skills and involved complex logic and understanding of relational data and because it only took you 30 minutes or an hour it must have been "easy". Or the way you are everybody's free tech advisor with family and friends... And things are said like "I'm not good with this stuff, but you're so good with it". For the record nobody is just "great" at technology or coding from birth its been a 2+ decade craft that I've experimented with and learned and put effort into. So taking into consideration all this effort I have put in to understand all this you say you'll never remember to push that button so you'll just ask me again when the problem arises. Yes because its so fun for me to constantly maintain your electronics because you can't bother to remember to push a button.5
It's essentially all meta you need to make a pokemon game, in csv files.
Afaik, they ripped the information from the original games, so you can be sure about their validity.
I love how it's easy to use, isn't some weird ass formatted wiki and even has scripts to load it into your database.
Me being a huge pokemon fan, that's the non plus ultra.
So last night i was dealing with some mysql query issues trying to grab some complex data from a relational database and ended up giving up.
This afternoon i was about half-asleep going to nap, not even thinking about the code and randomly came up with the solution.
A few hours later i made the changes and it works as intended.
Anyone else solve code subconsciously ? 😂😂😁😁12
We had a tool to analyze a distributed system. To do that it collected a large data set--basically the entire data set of the system itself, just in a different format--and ran analyses on that. That tool went through three iterations.
First: dump everything into an SQLite database and run SQL queries. That approach failed spectacularly when the tool was used on system larger than the toy examples used for sales demonstrations: if it ran at all, it ran for weeks. Which is not something clients are very happy about, usually.
Second: dump everything into many SQLite databases, one database for each type of thing. To avoid the evil, evil locking overhead of SQLite. Failed just as well as the first thing try, but it did run marginally faster.
Then it was dumped on my desk. With the instructions "make it fast". Nothing else. Ripped out the entire database code and rewrote it from scratch to still support relational queries (code reuse, yay!), then optimized the living hell out of that for an end result of 30dB improvement in runtime. At that point the tool was no longer the bottleneck, but I was still scolded for having changed the code in order to get there. Some people ...
Perhaps more of a wishlist than what I think will actually happen, but:
- Everyone realises that blockchain is nothing more than a tiny niche, and therefore everyone but a tiny niche shuts up about it.
- Starting a new JS framework every 2 seconds becomes a crime. Existing JS frameworks have a big war, until only one is left standing.
- Developing for "FaaS" (serverless, if I must use that name) type computing becomes a big thing.
- Relational database engines get to the point where special handling of "big data" isn't required anymore. Joins across billions of rows doesn't present an issue.
- Everyone wakes up one day and realises that Wordpress is a steaming pile of insecure cow dung. It's never used again, and burns in a fire.9
Store POJOs as json inside a database column so that we have a dynamic relational database!
For those who don't know what a POJO
*POJO: plain old Java object
Technically I was asked to store all data models as json in a single column 🙄10
Today at 'Derp & Co' a fellow co-worker decided that had duplicated data on Relational DB is good!
- Dev: 'but what if we have 2 companies in diferents groups?''
- co-worker: 'Just call it company A and Company B'
- Dev: 'but... this is not what...'
- co-worker: 'Trust me Dev, is the easy way'
I want the professional way, not the easy (and damn wrong) way :(, I can't improve myself like this.
Also, dead line is here too... TT^TT
Last sprint and still with doubts about the DB structure.12
My boi states that node's biggest issue has always been npm and the quality of packages. I always contradict those statements by saying that if one uses community standards and the best packages then one does not need to worry about the quality(i.e mongoose over some unmaintained mongo wrapper etc)
I sometimes catch myself finding that my way of thinking adapts better to JS than it even does Python (which is his preference for deep learning) and whilst there are some beastly packages for python in terms of quality and usefulness such as matplotlib etc that one can do great things with the equivalent JS.
I mean, tensorflow.js came from the same wizards that did tensorflow (obviously) and i find the functional approach of JS to be more on par with how we develop solutions.
I am no deep learning expert, and sadly I have no professional experience with machine learning. But I venture to say that we should not cast aside the great strides that the JS community has done to the language in terms of evolution and tooling. Today's Js is not your grandaddy's Js and thinking that the language is crippled because of early iterations of the language would be severely biased.
What do you guys(maybe someone with professional experience) think of Js as a language for machine learning?
Do you think the language poses something worth considering in terms of tooling and power for ml?3
For all the hate that Java gets, this *not rant* is to appreciate the Spring Boot/Cloud & Netty for without them I would not be half as productive as I am at my job.
Just to highlight a few of these life savers:
- Spring security: many features but I will just mention robust authorization out of the box
- Netflix Feign & Hystrix: easy circuit breaking & fallback pattern.
- Spring Data: consistent data access patterns & out of the box functionality regardless of the data source: eg relational & document dbs, redis etc with managed offerings integrations as well. The abstraction here is something to marvel at.
- Spring Boot Actuator: Out of the box health checks that check all integrations: Db, Redis, Mail,Disk, RabbitMQ etc which are crucial for Kubernetes readiness/liveness health checks.
- Spring Cloud Stream: Another abstraction for the messaging layer that decouples application logic from the binder ie could be kafka, rabbitmq etc
- SpringFox Swagger - Fantastic swagger documentation integration that allows always up to date API docs via annotations that can be converted to a swagger.yml if need be.
- Last but not least - Netty: Implementing secure non-blocking network applications is not trivial. This framework has made it easier for us to implement a protocol server on top of UDP using Java & all the support that comes with Spring.
For these & many more am grateful for Java & the big big community of devs that love & support it.
Spending hours upon hours trying to understand C# entity framework relational data mappings, going to the extreme of looking at LINQ queries, writing and rewriting over 100 lines of code only to find I needed to write the joins in dot notation on the include!5
There's very little good use cases for mongo, change my mind.
Prototyping maybe? Rails can prototype, create/update/destroy db schemas really quickly anyways.
If you're doing a web app, there's tons of libs that let you have a store in your app, even a fake mongo on the browser.
Are the reads fast? When I need that, use with redis.
Can it be an actual replacement for an app's db? No. Safety mechanisms that relational dbs have are pretty much must haves for a production level app.
Data type checks, null checks, foreign key checks, query checks.
All this robustness, this safety is something critical to maintain the data of an app sane.
Screw ups in the app layer affecting the data are a lot less visible and don't get noticed immediately (things like this can happen with relational dbs but are a lot less likely)
Let's not even get into mutating structures. Once you pick a structure with mongo, you're pretty much set.
Redoing a structure is manual, and you better have checks afterwards.
But at the same time, this is kind of a pro for mongo, since if there's variable data, as in some fields that are not always present, you don't need to create column for them, they just go into the data.
But you can have json columns in postgres too!
Is it easier to migrate than relational dbs? yes, but docker makes everything easy also.12
Nice to do some refactoring of the whole data access layer of our core logistics software, let me tell an story.
The project is around 80k lines of code, with a lot of integrations with an ERP system and an sql database.
The ERP system is old, shitty api for it also, only static methods through an wrapper to an c++ library
imagine an order table.
To access an order, you would first need to open the database by calling Api.Open(...file paths) (yes, it's an fucking flat file type database)
Now the database is open, now you would open the orders table with method Api.Table(int tableId) and in return you would get an integer value, the pointer.
Now for the actual order. first you need to search for it by setting the search parameter to the column ID of the order number while checking all calls for some BS error code
Api.SetInt(int pointer, int column, int query Value)
Then call the find method.
Then to top this shitcake of an api of: if it doesn't find your shit it will use the "close enough" method of search.
And now to read a singe string 😑
First you will look in the outdated and incorrect documentation given to you from the devil himself and look for the column ID to find the length of the column.
Then you create a string variable with ALL FUCKING SPACES.
Now you call the Api.GetStr(int pointer, int column, ref string emptyString, int length)
Now you have passed your poor string to the api's demon orgy by reference.
Then some more BS error code checking.
Now you have read an string value 😀
Now keep in mind to repeat these steps for all 300+ columns in the order table.
News from the creators: SQL server? yes, sql is good so everything will be better?
Now imagine the poor developers that got tasked to convert this shitcake to use a MS SQL server, that they did.
Now I can honestly say that I found the best SQL server benchmark tool. This sucker creams out just above ~105K sql statements per second on peak and ~15K per second for 1.5 second to read an order. 1.5 second to read less than 4 fucking kilobytes!
Right at that moment I released that our software would grind to an fucking halt before even thinking about starting it. And that me & myself and I would be tasked to fix it.
4 months later and two weeks until functional beta, here I am. We created our own api with the SQL server 😀
And the outcome of all this...
Fixes bugs older than a year, Forces rewriting part of code base. Forces removal of dirty fixes. allows proper unit and integration testing and even database testing with snapshot feature.
The whole ERP system could be replaced with ~10 lines of code (provided same relational structure) on the application while adding it to our own API library.
Best part is probably the performance improvements 😀. Up to 4500 times faster and 60 times less memory usage also with only managed memory.3
Has anyone here used MongoDB for relational databases? I am trying to use MySQL with Node but I feel like the whole thing is a mess. There are some ORMs like Knex or Sequelize for MySQL but they are pretty crap. I am thinking if I should just move onto MongoDB instead of MySQL, at the same time I feel a little dirty to store relational data in a document store :(18
First rant here...
Hand full of devs have to create a huge web platform that can shovel a lot of data around in about two months which is impossible...
Project lead has left major decisions in the hands of interns like database we want to use because no question can.be answered by that person. Inexperienced intern has chosen a fucking nosql database for highly relational datasets... why? Because new tech...
Development began and a bunch of problems arised... database was accessable from internet from day one. Random crashes because out of memory exceptions. Every possible feature had a description of at most 10 words... and no standards where enforced on anything.
Now that finaaaally we switch to sql after almost a year of prototypical production everybody keeps coding on new features so i have to port all the crap to the new database...
best part: a bunch of clients on different op systems have to be ported as well!
Even better part: i have to do that cause everybody else has practically no experience in any field...
And now the joke: i got hired for gui/desktop application development
Am i a wizard now?1
To me this is one of the most interesting topics. I always dream about creating the perfect programming class (not aimed at absolute beginners though, in the end there should be some usable software artifact), because I had to teach myself at least half of the skills I need everyday.
The goal of the class, which has at least to be a semester long, is to be able to create industry-ready software projects with a distributed architecture (i.e. client-server).
The important thing is to have a central theme over the whole class. Which means you should go through the software lifecycle at least once.
Let's say the class consists of 10 Units à ~3 hours (with breaks ofc) and takes place once a week, because that is the absolute minimum time to enable the students to do their homework.
1. Project setup, explanation of the whole toolchain. Init repositories, create SSH keys for github/bitbucket, git crash course (provide a cheat sheet).
Create a hello world web app with $framework. Run the web server, let the students poke around with it. Let them push their projects to their repositories.
The remainder of the lesson is for Q&A, technical problems and so on.
Homework: Read the docs of $framework. Do some commits, just alter the HTML & CSS a bit, give them your personal touch.
For the homework, provide a $chat channel/forum/mailing list or whatever for questions where not only the the teacher should help, but also the students help each other.
2. Setup of CI/Build automation. This is one of the hardest parts for the teacher/uni because the university must provide the necessary hardware for it, which costs money. But the students faces when they see that a push to master automatically triggers a build and deploys it to the right place where they can reach it from the web is priceless.
This is one recurring point over the whole course, as there will be more software artifacts beside the web app, which need to be added to the build process. I do not want to go deeper here, whether you use Jenkins, or Travis or whatev and Ansible or Puppet or whatev for automation. You probably have some docker container set up for this, because this is a very tedious task for initial setup, probably way out of proportion. But in the end there needs to be a running web service for every student which they can reach over a personal URL. Depending on the students interest on the topic it may be also better to setup this already before the first class starts and only introduce them to all the concepts in a theory block and do some more coding in the second half.
Homework: Use $framework to extend your web app. Make it a bit more user interactive with buttons, forms or the like. As we still have no backend here, you can output to alert or something.
3. Create a minimal backend with $backendFramework. Only to have something which speaks with the frontend so you can create API calls going back and forth. Also create a DB, relational or not. Discuss DB schema/model and answer student questions.
Homework: Create a form which gets transformed into JSON and sent to the backend, backend stores the user information in the DB and should also provide a query to view the entry.
4. Introduce mobile apps. As it would probably too much to introduce them both to iOS and Android, something like React Native (or whatever the most popular platform-agnostic framework is then) may come in handy. Do the same as with the minimal web app and add the build artifacts to CI. Also talk about getting software to the app/play store (a common question) and signing apps.
Homework: Use the view API call from the backend to show the data on the mobile. Play around with the mobile project to display it in a nice way.
5. Introduction to refactoring (yes, really), if we are really talking about JS here, mention things like typescript, flow, elm, reason and everything with types which compiles to JS. Types make it so much easier to refactor growing codebases and imho everybody should use it.
Flowtype would make it probably easier to get gradually introduced in the already existing codebase (and it plays nice with react native) but I want to be abstract here, so that is just a suggestion (and 100% typed languages such as ELM or Reason have so much nicer errors).
Also discuss other helpful tools like linters, formatters.
Homework: Introduce types to all your API calls and some important functions.
6. Introduction to (unit) tests. Similar as above.
Homework: Write a unit test for your form.
Our app has web scale data. So let me take all my tables from a relational database and map them one to one to a collection in a document database.
MongoDB database with really relational data. One main collection that had refs to four other collections, all of those references necessary to populate data for a page view. Complicated aggregate to populate all the necessary data and then filter based on criteria selected by the user. And then the client decides that he wants the information to be sortable by column. Some of those columns are fields on the main model, no problem. Others are fields on the refs, which is more of a problem. Especially given that these refs aren’t one single object. They’re arrays of objects.
The revelation was that I could just write an aggregate function to flat map the main collection, returning only the fields necessary for the search, and output it to a new collection and instead use that new collection for displaying and filtering/sorting search results.
But you can’t run the aggregate all the time, you surely say. If anything changes in the main collection, it won’t be reflected in the search results!
Mongoose post(‘findOneAndUpdate’) hooks, my friends. Mongoose post(‘findOneAndUpdate’) hooks.
Never been so happy to have a thing working properly in my life.2
After a year of using mongo in prod and personal projects I have realised some things. Its really nice early on the project, especially when there are changing requirements and for small projects or proof of concepts.
But when you make commercial software things tend to get more complex and relational. Stakeholders want reporting and even a report building which a document store isn't the best at.
With most projects projects when they get big things get relational and this becomes more and more expensive to handle in terms of compute power and developer time.
I don't doubt mongo has its place, maybe as an secondary specialised data store or if the project is inherently document oriented.
I've been dealing with a guy who knows almost nothing about tech and always insists on asinine systems and buzz words he's been sold by 3rd party vendors. Lately he really wants to replace our relational dbs with hdfs.
Our data is sub 10 gb in size.
I've been seriously considering making a contracting company and just preying on tech fools like this, I just don't know if I could ethically swallow it.
I had a discussion with my colleagues about my bachelor thesis.
Together we created within the last 18 month a REST-API where we use LDAP/LMDB as database (tree structured storage). Of course our data is relational and of course we have a high redundancy there. It's a 170 call API and I highly doubt that it's actually conforming REST.
Ensuring DB integrity is done in the backend and coding style there is "If we change it at one place, let's make sure to also change it everywhere else", so you get a good impression how much of spaghetti code we have there.
Now I proposed to code a solution in my bachelor thesis where we use a relational database (we even have an administrated Oracle DB with high availability) and have a write-only layer to also store the data in LDAP but my colleagues said that "it would add too much complexity to the system".
Instead I should write the relational layer myself and fetch the data somehow from the existing LDAP tree.
What the actual fuck, spaghetti code is what makes the system really unnecessarily complex so that no one will understand that code in 2 years.
Congratulations, you just created legacy code that went into production in 2018 while not accepting the opportunity to let that legacy code get eliminated.
Now good luck with running and maintaining that system and it's inconsistencies.1
Just notice that google spreadshit can do query like actual sql query.
Wtf each cell is like data as in relational databse.
Wtf wtf wtf.... soooooo cooool
My next db spreadsheet.6
Storing de-normalized data in NoSql when used to relational data feels like I am vomiting on the database
i don't like devs who put serialized data into a relational database. guys, do you even know how to relational database? 😭😭😭6
A guy who thought the only way to access data was through a relational data base and the only way to access a data base from java was JPA
Relational data is so much easier to work with than non-relational data (obviously!). You define the models, some serializers, some viewsets (set of generic views for GET, POST etc.) and their URL patterns, and you get REST APIs with full CRUD functionality. And if required each layer can be easily extended (provided the framework you're using is well-designed).
Best eays to improve database performance?
I have a huge database table with a lot of data, 10+columns and around 5-6million rows and growing. We are thinking about optimisation methods, the data will be frequently accessed, but not in the same ways. even though caching is helping, some initial requests take 6-20secs. What are good ways to improve a relational db performance? The data is date and item based, every item can have some data back from 2016 or older and up to 1000 related rows in each day. Every day new data would be added. Right now working with php.9