Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "graph databases"
-
Hey everyone,
We have a few pieces of news we're very excited to share with everyone today. Apologies for the long post, but there's a lot to cover!
First, as some of you might have already seen, we just launched the "subscribed" tab in the devRant app on iOS and Android. This feature shows you a feed of the most recent rant posts, likes, and comments from all of the people you subscribe to. This activity feed is updated in real-time (although you have to manually refresh it right now), so you can quickly see the latest activity. Additionally, the feed also shows recommended users (based on your tastes) that you might want to subscribe to. We think both of these aspects of the feed will greatly improve the devRant content discovery experience.
This new feature leads directly into this next announcement. Tim (@trogus) and I just launched a public SaaS API service that powers the features above (and can power many more use-cases across recommendations and activity feeds, with more to come). The service is called Pipeless (https://pipeless.io) and it is currently live (beta), and we encourage everyone to check it out. All feedback is greatly appreciated. It is called Pipeless because it removes the need to create complicated pipelines to power features/algorithms, by instead utilizing the flexibility of graph databases.
Pipeless was born out of the years of experience Tim and I have had working on devRant and from the desire we've seen from the community to have more insight into our technology. One of my favorite (and earliest) devRant memories is from around when we launched, and we instantly had many questions from the community about what tech stack we were using. That interest is what encouraged us to create the "about" page in the app that gives an overview of what technologies we use for devRant.
Since launch, the biggest technology powering devRant has always been our graph database. It's been fun discussing that technology with many of you. Now, we're excited to bring this technology to everyone in the form of a very simple REST API that you can use to quickly build projects that include real-time recommendations and activity feeds. Tim and I are really looking forward to hopefully seeing members of the community make really cool and unique things with the API.
Pipeless has a free plan where you get 75,000 API calls/month and 75,000 items stored. We think this is a solid amount of calls/storage to test out and even build cool projects/features with the API. Additionally, as a thanks for continued support, for devRant++ subscribers who were subscribed before this announcement was posted, we will give some bonus calls/data storage. If you'd like that special bonus, you can just let me know in the comments (as long as your devRant email is the same as Pipeless account email) or feel free to email me (david@hexicallabs.com).
Lastly, and also related, we think Pipeless is going to help us fulfill one of the biggest pieces of feedback we’ve heard from the community. Now, it is going to be our goal to open source the various components of devRant. Although there’s been a few reasons stated in the past for why we haven’t done that, one of the biggest reasons was always the highly proprietary and complicated nature of our backend storage systems. But now, with Pipeless, it will allow us to start moving data there, and then everyone has access to the same system/technology that is powering the devRant backend. The first step for this transition was building the new “subscribed” feed completely on top of Pipeless. We will be following up with more details about this open sourcing effort soon, and we’re very excited for it and we think the community will be too.
Anyway, thank you for reading this and we are really looking forward to everyone’s feedback and seeing what members of the community create with the service. If you’re looking for a very simple way to get started, we have a full sample dataset (1 click to import!) with a tutorial that Tim put together (https://docs.pipeless.io/docs/...) and a full dev portal/documentation (https://docs.pipeless.io).
Let us know if you have any questions and thanks everyone!
- David & Tim (@dfox & @trogus)53 -
Still trying to get good.
The requirements are forever shifting, and so do the applied paradigms.
I think the first layer is learning about each paradigm.
You learn 5-10 languages/technologies, get a feeling for procedural/functional/OOP programming. You mess around with some electronics engineering, write a bit of assembly. You write an ugly GTK program, an Android todo app, check how OpenGL works. You learn about relational models, about graph databases, time series storage and key value caches. You learn about networking and protocols. You void the warranty of all the devices in your house at some point. You develop preferences for languages and systems. For certain periods of time, you even become an insufferable fanboy who claims that all databases should be replaced by MongoDB, or all applications should be written in C# -- no exceptions in your mind are possible, because you found the Perfect Thing. Temporarily.
Eventually, you get to the second layer: Instead of being a champion for a single cause, you start to see patterns of applicability.
You might have grown to prefer serverless microservice architectures driven by pub/sub event busses, but realize that some MVC framework is probably more suitable for a 5-employee company. You realize that development is not just about picking the best language and best architecture -- It's about pros and cons for every situation. You start to value consistency over hard rules. You realize that even respected books about computer science can sometimes contain lies -- or represent solutions which are only applicable to "spherical cows in a vacuum".
Then you get to the third layer: Which is about orchestrating migrations between paradigms without creating a bigger mess.
Your company started with a tiny MVC webshop written in PHP. There are now 300 employees and a few million lines of code, the framework more often gets in the way than it helps, the database is terribly strained. Big rewrite? Gradual refactor? Introduce new languages within the company or stick with what people know? Educate people about paradigms which might be more suitable, but which will feel unfamiliar? What leads to a better product, someone who is experienced with PHP, or someone just learning to use Typescript?
All that theoretical knowledge about superior paradigms won't help you now -- No clean slates! You have to build a skyscraper city to replace a swamp village while keeping the economy running, together with builders who have no clue what concrete even looks like. You might think "I'll throw my superior engineering against this, no harm done if it doesn't stick", but 9 out of 10 times that will just end in a mix of concrete rubble, corpses and mud.
I think I'm somewhere between 2 and 3.
I think I have most of the important knowledge about a wide array of languages, technologies and architectures.
I think I know how to come to a conclusion about what to use in which scenario -- most of the time.
But dealing with a giant legacy mess, transforming things into something better, without creating an ugly amalgamation of old and new systems blended together into an even bigger abomination? Nah, I don't think I'm fully there yet.8 -
First year: intro to programming, basic data structures and algos, parallel programming, databases and a project to finish it. Homework should be kept track of via some version control. Should also be some calculus and linear algebra.
Second year:
Introduce more complex subjects such as programming paradigms, compilers and language theory, low level programming + logic design + basic processor design, logic for system verification, statistics and graph theory. Should also be a project with a company.
Year three:
Advanced algos, datastructures and algorithm analysis. Intro to Computer and data security. Optional courses in graphics programming, machine learning, compilers and automata, embedded systems etc. ends with a big project that goes in depth into a CS subject, not a regular software project in java basically.4 -
!Rant
Do any of you use graph databases. If so what? I am not able to understand where they really fit in.6 -
at one point in time, i had to work with a really junior backend team, they used javascript and neo4j as the database for an in-house developed community forum because "graph databases made sense" in the eyes of their tech lead
turns out that the team struggled quite a bit with it, and had some "unexpected complexity" problems when i asked them to add filters and sorting on the post endpoints
in the end, the "solution" they gave me was an endpoint that spewed ALL the posts so i could sort it in the front end
had they kept the same relational database they were using for the rest of the whole project, i'm quite sure it wouldn't take much to implement that (and their architecture was really performatic)
as a side project i rebuilt the whole forum in a weekend, but using postgresql as database, and it worked nicely, i even added some unit tests just for fun
gave myself a really big slap in the face after that, though1 -
So.
I just sat here and listened to some awful gibberish that sounded kind of like the language a person would use to describe logistics or construction, but that still lacked so much filler language that a straight spew of jargon doesn't seem likely.
reminds me of every single time I hear someone describe new technology that ends up bombing.
like the push towards graph databases which I personally can't understand the underlying storage mechanism which would make them work
of someone describing locks to your house that can be unlocked from a cellphoen over the internet.
or 2 form factor authentication and what happens if you lose your phone and there is no customer service ?
on that last maybe they could take a sample of every customers voice every year or a fingerprint or a blood sample :P1 -
I had a pretty good year! I've gone from being a totally unknown passionate web dev to a respected full stack dev. This will be a bit lengthy rant...
Best:
- Got my first full time employment dev role at a company after being self-taught for 8+ years at the start of the year. Finally got someone to take the risk of hiring someone who's "untested" and only done small and odd jobs professionally. This kickstarted my career, super grateful for that!
- Started my own programming consulting company.
- Gained enough confidence to apply to other jobs, snatched a few consulting jobs, nailed the interviews even though I never practiced any leet code.
- Currently work as a 99% remote dev (only meet up in person during the initialization of some projects.) I never thought working remotely could actually work this well. I am able to stay productive and actually focus on the work instead of living up to the 9-5 standard. If I want to go for a walk to think I can do that, I can be as social and asocial as I want. I like to sleep in and work during the night with a cup of tea in the dark and it's not an issue! I really like the freedom and I feel like I've never been more productive.
- Ended up with very happy customers and now got a steady amount of jobs rolling in and contracts are being extended.
- I learned a lot, specialized in graph databases, no more db modelling hell. Loving it!
- Got a job where I can use my favorite tools and actually create something from scratch which includes a lot of different fields. I am really happy I can use all my skills and learn new things along the way, like data analysis, databricks, hadoop, data ingesting, centralised auth like promerium and centralised logging.
- I also learned how important softskills are, I've learned to understand my clients needs and how to both communicate both as a developer and an entrepeneur.
Worst:
- First job had a manager which just gave me the specifications solo project and didn't check in or meet me for 8 weeks with vague specifications. Turns out the manager was super biased on how to write code and wanted to micromanage every aspect while still being totally absent. They got mad that I had used AJAX for requests as that was a "waste of time".
- I learned the harsh reality of working as a contractor in the US from a foreign country. Worked on an "indefinite" contract, suddenly got a 2 day notification to sum up my work (not related to my performance) after being there for 7+ months.
- I really don't like the current industry standard when it comes to developing websites (I mostly work in node.js), I like working with static websites (with static website generators like what the Svelte.js driver) and use a REST API for dynamic content. When working on the backend there's a library for everything and I've wasted so many hours this year to fix bugs and create workarounds related to dependencies. You need to dive into a rabbit hole for every tool and do something which may work or break something later. I've had so many issues with CICD and deployment to the cloud. There's a library for everything but there's so many that it's impossible to learn about the edge cases of everything. Doesn't help that everything is abstracted away, which works 90% of the time but I use 15 times the time to debug things when a bug appears. I work against a black box which may or may not have an up to date documentation and it's so complex that it will require you to yell incantations from the F#$K
era and sacrifice a goat for it to work properly.
- Learned that a lot of companies call their complex services "microservices". Ah yes, the microservice with 20 endpoints which all do completely unrelated tasks? -
Some friends of mine were working on doing neural network image processing and wanted to build a social network for it. I got to play with graph databases, mobile app development, and neural nets. Unfortunately, project never took off, but it was fun nevertheless.
-
I don't know about tinkepop but neo4j is the slowest fucking thing I have ever ran on my desktop, their fucking java based browser included.
and btw THAT is the reason I down Java apps, they're fucking slow most of the time !
https://saashub.com/compare-apache-...
Is that an aspect of graph databases ? slowness ?
according this guy yes.
https://memgraph.com/blog/...
except for specific tree traversals which makes me wonder what kind of things theyd be good for. -
Not a rant, but seeking advice...
Should I abandon 2 years' worth of work on migrating a personal project from SQL (M$) to a Graph database, and just stick to SQL? And only consider migrating when/if I need graph capabilities?
The project is a small social media platform. Has around ~50 monthly active users.
Why I started the migration in the first place:
• When researching databases, I read that for social media, graph is more suitable. It was, at least in terms of query structure. It was more natural, there were no "joins", and queries were much simpler than their SQL counterparts.
• In case the project got big, I didn't want to have to panic-deal with database issues that come with growth. I had some indexing issues with MSSQL, and it got me worried that at 50MAU I'm having these issues, what would happen if I get more?
• It's a personal project, and the Gremlin language and graph databases looked cool and I was motivated to learn something new.
----
Why I'm considering aborting the migration:
• It's taking too damn long. I'm unable to work on other features because this migration is taking up all my free time. Sunk cost fallacy is hitting me hard with this one.
• In local testing within docker, it's extremely slow. I tried various graph engines (janusgraph, official tinkerpop, orientdb), and the fastest one takes 4-6minutes to complete my server tests. SQL finishes the same tests in under 2 minutes, same docker environment. I also tried running my tests on a remote server (AWS neptune) and it was just as slow. Maybe my queries are bad, but can I afford to spend even more time fine tuning all queries?
• I now realise that "graph = no scalability issues" was naïve of me, and 100% wishful thinking. Scalability issues don't care what database I use, but about how well tuned and configured the whole system is.
• I really want to move on. My tech stack is falling behind and becoming outdated. I'm unable to maintain dependencies.
• I'm worried about losing those 50 MAU because they're essential to gaining traction once I release the platform. I keep telling them about the migration but at some point (2 years later) they're going to get bored I feel.
I guess partially it's a rant because I feel like I shouldn't stop now having spent 2 years on this, but at the same time I feel like I'm heading towards a dead end.
If you made it this far, thank you for reading:)10