24
vocuzi
6y

Client : We need real time analysis.

Me : But we can't just scrape thousands of results and process them on user's click.

Client : Don't do that, Real-time analysis is scraping it once and processing it everytime the user demands.

Me : Okay

WHAT THE FUCK !!!!!

Comments
  • 2
    The client is right.
    The data can be stored once (you can even make a cron job to scrape on a given timeframe like once a day) and the results are triggered by an event and they are given in real time by a process that iterates, analyzes and/or filters the data.

    I see no problem with that
  • 2
    @bioDan that's what I'm doing, but is this realtime ?
  • 3
    @vocuzi the results are. the data isn't.

    I suspect you are doing something with big data and AI (ML/DL) ?
  • 3
    That's not real time analysis that's instant result on user click. just my two cents
  • 1
    @bioDan how come output is realtime without being relative to data ?
  • 2
  • 3
    @gitpush well.. in the classic sense of a machine connected to a power generator feeding real-time data to a monitoring software, you are right. Real-time refers to the currency of the data.

    but I assume @vocuzi is talking about big data analysis and a.f.a.i.k performing a simple 'click' to analyze the data can take hours, days or seconds depends on the trade-offs between cost and performance.

    Therefore providing results analyzing/filtering big data can also be referred as 'real-time' in terms of getting the end-result quickly (about 5 seconds ~ totally made this number up). But that might just be a sales pitch I heard so I'm not sure, I'm not an expert in the field (yet!).
Add Comment