22
nurked
71d

Boss: I need to analyze this data right away! Here is the log in CSV file, find out what happened. Who did what! Load it into Excel and give me graphs of that and that.

Me: Excel? Sir, with all due respect - I would not allow you to touch those disgusting excel graphs! I know better solution! (Stand up and walk into a phone booth, quickly changing)

--2 seconds later--

Elasticshearchman: It's me! I heard you need to analyze some data! Let me help you!

Boss: *Facepalm* Again? Please, you are using too much of that elastic. Can you stop? The only thing I need is a couple graphs based on our log data.

Elasticsearchman: But Sir, just one more time, it will be awesome!

Boss: alright, alright... But make sure I'll get them by noon.

Elasticsearchman: Get ready to be amazed!

--2 minutes later--

Elasticsearchman: Ok, let's see what do we have here? My laptop is at home. And my computer in the other building. Alright, here is one station connected to the domain. And I'm not an admin there. Let's start from here. I just need to run one python script to load everything to elastic.

No python.

Ok. Can I hit my elastic? No, port block.

Ok, now what? Emmm. I can ssh to one random nginx server and in there. Forward proxy - looks like it's working! YES! Elastic is online. And Kibana is here!

Now what?

Now to upload 30000 records. Ok, I need python. Nope. Nodejs? Nope. C#? Nope. Powershell? Eeeh, i'm too rusty for that. CMD? Oh, I'm not trying to torture myself!

Yes! Elasticsearchie-feeling tells me that elastic supports bulk load. For that I just need to generate a simple json file.

Generate how? If only I could...

Jump up and run to the phone booth, change quickly.

--2 seconds later--

Excel-formulaman: I'm an Excel-formulaman everyone! You want to convert your csv to json? No problem!!!

Here is your solution (see picture at the end)

--2 hours later--
Boss: You know, your graphs are really cool.
Me: Yes, requires a lot of elaborate coding, but this pays off.
Boss: Yes, I'll call Elasticsearchman more often in the future.
Me: *Facepalm*

I think I need to find a better ingest mechanism than that.

Comments
  • 6
    import-csv FILE | convertto-json
  • 3
    @stop *Facepalm* yet again.

    Alright, so this command has been running for the last 2 hours trying to convert 185mb CSV file into JSON on my laptop. Who will win? Command or Laptop? Disk, Memory and Processor are at 100%
  • 9
  • 2
    @stop If you are interested... It has been like 4 hours and my laptop still fighting that conversion process. I'm thinking I would reboot it in about 2 hours if it's not done...
  • 0
    What's wrong with logstash?
    While it's fully capable of many other things as well, outputing to ES is literally the reason it exists. And it handles CSV very well in my experiences.
  • 2
    @deadPix3l

    I was too far away from my docker farm to spin up another logstash thing. It's logstashing away now...
  • 2
    @devTea it's weekend give me a break 😭
    I don't like to work on weekends 🤷🏼‍♂️😪😛
  • 6
    @gitpush ........ no
  • 3
    @devTea this is my auto reply on weekends 🤷🏼‍♂️
  • 2
    @gitpush wow... I think it's a second "no words" I earned from you!
  • 5
    @gitpush see people take your quote as achivement

    Lol
  • 2
    @devTea Hold on... Isn't it some kind of an achievement?
Your Job Suck?
Get a Better Job
Add Comment