So my customer wanted me to collect Big Data using SQLITE... (Yeap SQLITE, like wtf...)

F*** my career.

"I need you to dig a ditch. Here: use this spoon..."

@xMadxHatterx. There's a popular saying in the matrix , " There's no spoon" ....

Is it safe big data though? It's a bit of a buzzword at the mo. I've spoken to people who have "big data" projects, and it turns out they have a few hundred customer records.

@AlmondSauce Another thing to hate about job postings, more concretely, "big data developer". All these big deal titles only serve to scare the recruits they needed.

@AlmondSauce just 100? That's not big data...

@johnmelodyme Precisely...

Well you can collect up to 281 terabyte so it can be done. Don't say you should but you could probably make it work in certain situations.

@AlmondSauce Big data is not only measured in terms of number of rows.
The number of columns and amounts of relationships in the data are also sufficient to categorize a data warehouse/datalake as big data.

That being said, it sounds like you are right and its not big data.

Also, big data is more than a buzz-word, even if its being overly misused and overused.

@bioDan Sure, but it's a good *basic* guide at least. I've never seen anything with a few hundred rows that can come close to being labelled as big data.

@johnmelodyme persuade the client to use a more suitable database.

Tell him its not scalable and its as if he is asking you to participate in a car race with a tricycle and win, and then keep on participating in car races.

@AlmondSauce well you may have 100 rows and 120K columns of different datatypes in which some of them store logs or binary objects up to 20 MB.

@bioDan Have you ever seen that in practice?! Genuinely curious as I've never come across a system that would store data that way.

@AlmondSauce yup, data lakes and in general column-based dbs.

In one of them, each row represented a client of the company, and their entire database, logs, and version tracking were spreaded as columns

@bioDan Interesting case, thanks for sharing 👍