Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "derived data"
Woohoo!!! I made it to 1000++s :) Now I feel less newbie-like around here :)
So... I don't want to shit-post, so in gratitude to all you guys for this awesome community you've built, specially @trogus and @dfox, I'll post here a list of my ideas/projects for the future, so you guys can have something to talk about or at least laugh at.
Here we go!
Current Project: Ensayador.
It's a webapp that intends to ease and help students write essays. I'm making it with history students in mind, but it should also help in other discipline's essay production. It will store the thesis, arguments, keywords and bibliography so students can create a guideline before the moment of writting. It will also let students catalogue their reads with the same fields they'd use for an essay: that is thesis, arguments, keywords and bibliography, for their further use in other essays. The bibliography field will consist on foreign keys to reads catalogued. The idea is to build upon the models natural/logical relations.
Apps: All the apps that will come next could be integrated in just one big app that I would call "ChatPo" ("Po" is a contextual word we use in my country when we end sentences, I think it derived from "Pues"). But I guess it's better to think about them as different apps, just so I don't find myself lost in a neverending side-project.
A subchat(similar to a subreddit)-based chat app:
An app where people can join/create sub-chats where they can talk about things they are interested in. In my country, this is normally done by facebook groups making a whatsapp group and posting the link in the group, but I think that an integrated app would let people find/create/join groups more easily. I'm not sure if this should work with nicknames or real names and phone numbers, but let's save that for the future.
A slack clone:
Yes, you read it right. I want to make a slack clone. You see, in my country, enterprise communications are shitty as hell: everything consists in emails and informal whatsapp groups. Slack solves all these problems, but nobody even knows what it is over here. I think a more localized solution would be perfect to fill this void, and it would be cool to make it myself (with a team of friends of course), and hopefully profit out of it.
A labour chat-app marketplace:
This is a big hybrid I'd like to make based on the premise of contracting services on a reliable manner and paying through the app. "Are you in need of a plumber, but don't know where to find a reliable one? Maybe you want a new look on your wall, but don't want to paint it yourself? Don't worry, we got you covered. In <Insert app name> you can find a professional perfect to suit your needs. Payment? It's just a tap away!". I guess you get the idea. I think wechat made something like this, I wonder how it worked out.
* Why so many chat apps? Well... I want to learn Erlang, it is something close to mythical to me, and it's perfect for the backend of a comms app. So I want to learn it and put it in practice in any of these ideas.*
Flat-land arena: A top down arena game based on the book "flat land". Different symmetrical shapes will fight on a 2d plane of existence, having different rotating and moving speeds, and attack mechanics. For example, the triangle could have a "lance" on the front, making it agressive but leaving the rest defenseless. The field of view will be small, but there'll be a 2d POV all around the screen, which will consist on a line that fills with the colors of surrounding objects, scaling from dark colors to lighter colors to give a sense of distance.
This read could help understand the concept better:
A 2D darksouls-like class based adventure: I've thought very little about this, but it's a project I'm considering to build with my brothers. I hope we can make it.
Imposible/distant future projects:
History-reading AI: History is best teached when you start from a linguistic approach. That is, you first teach both the disciplinar vocabulary and the propper keywords, and from that you build on causality's logic. It would be cool to make an AI recognize keywords and disciplinary vocabulary to make sense of historical texts and maybe reformat them into another text/platform/database. (this is very close to the next idea)
Extensive Historical DB: A database containing the most historical phenomena posible, which is crazy, I know. It would be a neverending iterative software in which, through historical documents, it would store historical process, events, dates, figures, etc. All this would then be presented in a webapp in which you could query historical data and it would return it in a wikipedia like manner, but much more concize and prioritized, with links to documents about the data requested. This could be automated to an extent by History-reading AI.
I'm out of characters, but this was fun. Plus, I don't want this to be any more cringy than it already is.12
This is either a shower thought or a sober weed thought, not really sure which, but I've given some serious consideration to "team composition" and "working condition" as a facet of employment, particularly in regard to how they translate into hiring decisions and team composition.
I've put together a number of teams over the years, and in almost every case I've had to abide by an assemblage of pre-defined contexts that dictated the terms of the team working arrangement:
1. a team structure dictated to me
2. a working temporality scheme dictated to me
3. a geographic region in which I was allowed to hire
4. a headcount, position tuple I was required to abide by
I've come to regard these structures as weaknesses. It's a bit like the project management triangle in which you choose 1-2 from a list of inadequate options. Sometimes this is grounded in business reality, but more often than not it's because the people surrounding the decisions thrive on risk mitigation frameworks that become trickle down failure as they impose themselves on all aspects of the business regardless of compatibility.
At the moment, I'm in another startup that I have significantly more control over and again have found my partners discussing the imposition of structure and framework around how, where, why, who and what work people do before contact with any action. My mind is screaming at me to pull the cord, as much as I hate the expression. This stems from a single thought:
"Hierarchy and structure should arise from an understanding of a problem domain"
As engineers we develop processes based on logic; it's our job, it's what we do. Logic operates on data derived from from experiments, so in the absence of the real we perform thought experiments that attempt to reveal some fundamental fact we can use to make a determination.
In this instance we can ask ourselves the question, "what works?" The question can have a number contexts: people, effort required, time, pay, need, skills, regulation, schedule. These things in isolation all have a relative importance ( a weight ), and they can relatively expose limits of mutual exclusivity (pay > budget, skills < need, schedule < (people * time/effort)). The pre-imposed frameworks in that light are just generic attempts to abstract away those concerns based on pre-existing knowledge. There's a chance they're fine, and just generally misunderstood or misapplied; there's also a chance they're insufficient in the face of change.
Fictional entities like the "A Team," comprise a group of humans whose skills are mutually compatible, and achieve synergy by random chance. Since real life doesn't work on movie/comic book logic, it's easy to dismiss the seed of possibility there, that an organic structure can naturally evolve to function beyond its basic parts due to a natural compatibility that wasn't necessarily statistically quantifiable (par-entropic).
I'm definitely not proposing that, nor do I subscribe to the 10x ninja founders are ideal theory. Moreso, this line of reasoning leads me to the thought that team composition can be grown organically based on an acceptance of a few observed truths about shipping products:
1. demand is constant
2. skills can either be bought or developed
3. the requirement for skills grows linearly
4. hierarchy limits the potential for flexibility
5. a team's technically proficiency over time should lead to a non-linear relationship relationship between headcount and growth
Given that, I can devise a heuristic, organic framework for growing a team:
- Don't impose reporting structure before it has value (you don't have to flatten a hierarchy that doesn't exist)
- crush silos before they arise
- Identify needed skills based on objectives
- base salary projections on need, not available capital
- Hire to fill skills gap, be open to training since you have to pay for it either way
- Timelines should always account for skills gap and training efforts
- Assume churn will happen based on team dynamics
- Where someone is doesn't matter so long as it's legal. Time zones are only a problem if you make them one.
- Understand that the needs of a team are relative to a given project, so cookie cutter team composition and project management won't work in software
- Accept that failure is always a risk
- operate with the assumption that teams that are skilled, empowered and motivated are more likely to succeed.
- Culture fit is a per team thing, if the team hates each other they won't work well no matter how much time and money you throw at it
Last thing isn't derived from the train of thought, just things I feel are true:
- Training and headcount is an investment that grows linearly over time, but can have exponential value. Retain people, not services.
- "you build it, you run it" will result in happier customers, faster pivoting. Don't adopt an application maintenance strategy
WASTED MY ENTIRE DAY AJD ENTIRE NIGHT.
MY flutter app isn’t running on iPhone simulator!!! I cleaned and deleted derived data and cleaned thousand times but it crashes at startup without error! What the fuckkkkk do I do!!????
Oh and my iPhone is not being recognised by any IDE Idk why. :((((((((( pls m so miserable pls dnt :(4
Persisterising derived values. Often a necessary evil for optimisation or privacy while conflicting with concerns such as auditing.
Password hashing is the common example of a case considered necessary to cover security concerns.
Also often a mistake to store derived values. Some times it can be annoying. Sometimes it can be data loss. Derived values often require careful maintenance otherwise the actual comments in your database for a page is 10 but the stored value for the page record is 9. This becomes very important when dealing with money where eventual consistency might not be enough.
Annoying is when given a and b then c = a + b only b and c are stored so you often have to run things backwards.
Given any processing pipeline such as A -> B -> C with A being original and C final then you technically only need C. This applies to anything.
However, not all steps stay or deflate. Sum of values is an example of deflate. Mapping values is an example of stay. Combining all possible value pairs is inflate, IE, N * N and tends to represent the true termination point for a pipeline as to what can be persisted.
I've quite often seen people exclude original. Some amount of lossy can be alright if it's genuine noise and one way if serving some purpose.
If A is O(N) and C reduces to O(1) then it can seem to make sense to store only C until someone also wants B -> D as well. Technically speaking A is all you ever need to persist to cater to all dependencies.
I've seen every kind of mess with processing chains. People persisting the inflations while still being lossy. Giant chains linear chains where instead items should rely on a common ancestor. Things being applied to only be unapplied. Yes ABCBDBEBCF etc then truncating A happens.
Extreme care needs to be taken with data and future proofing. Excess data you can remove. Missing code can be added. Data however once its gone its gone and your bug is forever.
This doesn't seem to enter the minds of many developers who don't reconcile their execution or processing graphs with entry points, exist points, edge direction, size, persistence, etc.2