While reading through the Elasticsearch (Java search engine) source code a while ago I found this gem:

return i == -1? -1: i;

I think someone should stop drinking while coding.

Some other nice lines:

int i = 0;
return j + 1000 * i;

Are these guys high?

Having to implement full text and filter search on JSON data using ElasticSearch on a Django app within 24 hours...

Sure client. Should I also insert this flaming metal cactus inside of myself?

There's so many cool technologies too look into now a days, but so little time.
-docker
-laravel
-.net
-react
-es6/es7
-elasticsearch
-redis
-unit testing

Where do i download more hours to add to my days?

<just got out of this meeting>
Mgr: “Can we log the messages coming from the services?”
Me: “Absolutely, but it could be a lot of network traffic and create a lot of noise. I’m not sure if our current logging infrastructure is the right fit for this.”
Senior Dev: “We could use Log4Net. That will take care of the logging.”
Mgr: “Log4Net?…Yea…I’ve heard of it…Great, make it happen.”
Me: “Um…Log4Net is just the client library, I’m talking about the back-end, where the data is logged. For this issue, we want to make sure the data we’re logging is as concise as possible. We don’t want to cause a bottleneck inside the service logging informational messages.”
Mgr: “Oh, no, absolutely not, but I don’t know the right answer, which is why I’ll let you two figure it out.”
Senior Dev: “Log4Net will take care of any threading issues we have with logging. It’ll work.”
Me: “Um..I’m sure…but we need to figure out what we need to log before we decide how we’re logging it.”
Senior Dev: “Yea, but if we log to SQL database, it will scale just fine.”
Mgr: “A SQL database? For logging? That seems excessive.”
Senior Dev: “No, not really. Log4Net takes care of all the details.”
Me: “That’s not going to happen. We’re not going to set up an entire sql database infrastructure to log data.”
Senior Dev: “Yea…probably right. We could use ElasticSearch or even Redis. Those are lightweight.”
Mgr: “Oh..yea…I’ve heard good things about Redis.”
Senior Dev: “Yea, and it runs on Linux and Linux is free.”
Mgr: “I like free, but I’m late for another meeting…you guys figure it out and let me know.”
<mgr leaves>
Me: “So..Linux…um…know anything about administrating Redis on Linux?”
Senior Dev: ”Oh no…not a clue.”

It was all I could do from doing physical harm to another human being.
I really hate people playing buzzword bingo with projects I’m responsible for.

Only good piece is he’s not changing any of the code.

Manager: let's use elasticsearch for performing relational queries. PostgreSQL performance is not great.

Me: Say what? 👿

Elasticsearch 404 page.
**Puff of logic**

Has anyone installed Elasticsearch on Linux - centos to be specific.

Trying to workout why the fucker won't install. Setting up a proof of concept so don't want to use it currently as SaaS.

From why I can tell, it only needs Java, (check) and to be ran as a user other then root (check) but running ./bin/Elasticsearch hangs after a while and starts powering up 100 odd threads with no progress.

Elasticsearch queries are FUCKING ugly

Elasticsearch documentation is FUCKING bare-minimum.

Kibana only shows data when it FUCKING feels like it.

Elastic stack is FUCKING annoying me

They ask weird questions...
I give weird answers...
Why else would I join a QnA forum?

My boss did not care about making things secure in our early development stage, even though I told him several times.
After 1 day our elastic search cluster was filled with random crappy data.

Fix: Apply security schemes provided by AWS

Today marks the day that i finally get to do stuff on a production server.
Its just installing the elasticsearch cluster. But i still feel honored by the trust im given even tho im still an apprentice.

kimchy ≒ kimchi (????)

Kimchi : A traditional fermented Korean dish made of vegetables with varied seasonings.

I wonder what Elasticsearch meant to say. LOL

Job frustrated me again today.

The shit just keeps on commiting suicide...

Cannot talk much about it, but essentially it's faulty software killing randomly one or one up to N servers running elasticsearch...

Conversation between me and a good friend:

Me: No gaming today, work todo.
Him: noooo...
Me: Yes...... Could u go buy some groceries? Household help is sick.
Him: maybe...what u need?
Me: coffee. I need frigging fucking coffee.
Him: ok. How bad is it.......
Me: empty today.
Him: will be at your house in hour. DON'T DO ANYTHING STUPID.

...

It's funny how good friends immediately sense danger and become very attentive when the lack of coffee and myself is mentioned in one sentence.

Note to self: Pointing your tests at a non disposable DB will cause very very bad things to happen. No idea what the flying fuck I was thinking - but praise to the data gods it wasn't a production elastic!

Am I the only one who just loves when I do "curl 127.0.0.1:9200/_template" and get back entire 32" screenful of compressed JSON?

Frikin elastic...

Yes, I’m mr. not provided. 🙃

3 weeks.

Am I just stupid? It took me 3 whole weeks to finally come to the realisation today that the Elasticsearch "guide" and Elasticsearch "reference" were different, with different version numbers. I've been ignoring Google search results that say Elasticsearch 2.x for WEEKS and wondering why I couldn't find a solution to simple problems.

Turns out, the current Elasticsearch "guide" is on version 2.x while Elasticsearch itself is on version 6.x.

They even have almost identical URLs that go ../guide/../reference and ../guide/../guide.

WHY? Why would you do that? Am I just stupid? Am I still getting it wrong? What the heck is up with Elasticsearch documentation?

So I was setting up ELK (Elasticsearch, Logstash and Kibana) all in one EC2 on AWS today for demo purposes. I had everything prepared. Elastic IP, correct security group rules, etc.

I figured I would just do quick test before writing filters and templates if I can access Kibana. So I started service for it and tried to open it with Chrome.

Timeout.

Checked config file. Compared it to documentation. Seemed good but changed some things just for sake of change. Restarted service.

Timeout.

Reverted changes I've made in config. Restarted service. Curl on localhost. It work... OK. 😐

It took me half an hour but finally I figured it out after I took my phone and opened it from there. It was working from the beginning. Stupid company network was for some reason blocking this connection. Fuck! 😡And I was restarting that poor service like crazy trying to fix something that wasn't broken.

I ran elasticsearch reindexing on production.

My manager asked why there's no item shown on the search page, and I slowly told him, I ran reindex on production.

💣💣💣

This is so annoying, I had 9 diff. jobs the past 2 years and this is my 10th and if this doesn't change I might reconsider my options again.

I came to work at a company that pays me like a Junior and treats me as an intern. My 20yo "boss" who acts as a project owner/lead dev doesn't want to learn anything new and sees any improvement as a waste of money. The problem is he thinks hes a great programmer but he doesn't know shit. Im mainly working on the Laravel installation because "I claimed I know Laravel". And its absolute garbage. They haven't used a single Laravel features besides routes and everything else is vanilla PHP. They write for loops that loop through $_REQUEST to remove a single character. Write 100 deep nested ifs and they abuse Elasticsearch to the point ES crashes because the program is using 1000 deep multidimensional arrays. Its only a webshop...

Everytime I try to make a suggestion like making the master branch protected, doing code reviews etc etc I get shut down because they are autistic and don't want anything to change.

I just gave a simple API which fetches recent searches by pinging an index on elasticsearch to the UI developer.
She had just one job. And ended up calling the server every time it loads on every screen thereby reaching max limit of calls per second and giving 429.

QA are not even required to break your code. UI developers are more than enough :)

So, since almost a week I was trying to get familiar with Algolia which was to be used as API for the search feature in our App. But now, we are going with Elastic Search.

So this web company i joined had a page load time in minutes. The free text search (inverted index search, based on elasticsearch) queries would return results in 10-45 seconds (should be milliseconds always). The indexes had no schema. And they would crawl data and feed into mssql db, which had a 2 gb/db limit on the free version. So everytime the db hit the limit, a new db was created and the name was incremented by one.

Had a very tough time cleaning up that mess. Plus the architect who had made this architecture was on his way out and unhelpful to the core.

What was worse was that most of the changes i did were very simple changes that should have been done long back. Basic sanity changes.

Manager said we need to use Queue. Several meetings after then I looked at prototype by 6 senior devs:

A QueueListener connects to RabbitMQ check for payload then *disconnects*;

A TaskProvider in ASP.Net.MVC.Core(whatever it is) listening http and dependency inject that QueuePoller;

A Visual Cron timer calls that http url every 5 minutes.

Wait for it: a set of database tables to store messages for another MessageProcessor.

It’s a XML to CSV file conversion project consists of 43 unique projects under a solution. I did it within 500 lines of Node with ElasticSearch and told we don’t use fancy new stuffs here.

Elasticsearch.

You can't have distributed free text search and not have elasticsearch in the same sentence.

A lot of analytics companies are running because of the elasticsearch aggregation framework. And search couldn't have been faster on such mass of data.

P.S. i used to be a solr fanboy, then i met elasticsearch. Kimchy knows the best.

So a few days ago I sat down to write a redis adaptor to transfer data back and forth between redis and elasticsearch. I download the go-redis package and start writing a simple client.

I run the client and it gives me an error. So I'm stuck at it for about 30 mins and then I say to myself, "You dumb fuck you haven't started the redis-server". So I open up another terminal and type in `redis-server` and then I realise I don't even have redis installed on my machine.

I do such dumb things every weekend. If you have any dumb mistakes you made while writing code please share them in the comments. :-)

When elastic search teaches you life lessons

Tomcat Loading Time

The 'lead' developer is unable to comprehend why sending an empty string when it doesn't exist (instead of not sending it at all and setting it later when it becomes available) is not the best idea to do. Instead, everything is the fault of ElasticSearch (which I oversee in some capacity) because it doesn't read stupid! And so any error being caused is due to ES. YOU DENSE MOTHERFUCKER!!! FUUUUUUUUUUUUUUUUUUUUUUUUUUUU

When I got to know about elasticsearch. Just put the data and search whatever you want

Dear AWS, your Elasticsearch service is a bogus pile of shit-engorged horse fly larvae. Not only do you give no useful visibility into what's happening with the cluster (making diagnosis a sadistic guessing game), you lock down the fucking settings API, making it impossible to debug!! But your excellent support is on it! I wonder if I'll hear back from them this week with another inane suggestion like "increase the node count". Meanwhile the rest of my system is limping along, sometimes getting data where it's supposed to be while I keep fake-smiling and reassuring management and customers that "I'm working on it". If you're going to offer a service either make sure it works or get the fuck out of my way. I'll be moving my cluster back to EC2 and you can go do a back flip off a skyscraper. I need a drink

Having to do 10's of ElasticSearch v2 -> 6 client requests migration manual deploys of a 10-year old PHP/ Javascript codebase where the gulp build's plugins no longer work, the package dependencies have been .gitignore'd (who does that!), and dev cowboys have frequently bypassed version control by making changes directly in production.

Also, no one knows anything about it because the only dev who was supposed to maintain this app left 3 months ago due to unbearable management.

Fuck Redis and ElasticSearch for being so utterly goddamn useless.

So I've been given a task to monitor a whole lot of logs of some servers (whole university ~ 10+ departments). The technologies are diverse so I'm cramming everything into elasticsearch via logstash (and filebeat), viewing it into kibana. Any recommendations for what should be the 'useful' stuff to be viewed into dashboard? I guess:
- Overall traffic wtih respect to previous days/weeks
- Most viewed domains
- 200
- 404
- 503
- Failed logins?
- Dropped connections?
- Critical-load of systems? 90%+

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Today I wrote my first plugin ifor elasticsearch ... Was awsome feeling..it is cool to decorate the readme when you have written something of your own.

i feel its a great time to be a developer we have so many toys to play with

machine learning, scientific python, nodejs, frontend js frameworks, nosql, NLP, elasticsearch, mongodb, open source .net, big data with java, arduino..., VR, 3d printing

what toys are you playing with?

So, there were four judgement rounds, over a period of 36 hours.

During the 3rd judgement, the judge says we have a potentially winning project, we just need to put things together now.

During the fourth judgement round, my laptop's Network Interface Card crashes, while running Node server and ElasticSearch server (while another laptop was running a Django server)...

On top of that, the judge assumes that the probability distribution of having a chest disease that we were showing in the form of heatmap on a chest X-ray, was actually body heatmap... And we were saying wherever there is more heat, is the diseased part.

My only hackathon...

Hi, I am using a Wikipedia scrapper in one of my Open Source project. The data extracted from it is the stored in Elasticsearch... Now I have decided to create library out of it so that other people can use it too... My question is should also include the Elasticsearch storing module in library or just add the scrapper... Please let me know your thoughts.

I've a whole new respect for ElasticSearch. It's codebase is so insanely complex, that I'm seriously contemplating tracing out the flow on a big ass chart. Any suggestions on how you people work and debug so many asynchronous flows?

I have been working on a bug, for almost 6 days (to be read as 3 consecutive weekends), and the best I've done is, conceptually isolate where it's happening. I'm an open source noob, but I feel I've learnt a whole lot during sifting through ES' codebase. :)

Found a bug today that made me groan in frustration.

It appears that the official elasticsearch debian package checks if the system's init daemon is systemd by... Checking if systemctl binary is available.

Issue is... Systems might contain that binary while using a different init, as the binary is part of the "systemd" package.

To actually switch to systemd however, the package systemd-sysv has to be installed, which creates a link from /bin/init to systemd's main executable.

What happens when your system doesnt use systemd then? The postinstall/preremove scripts fail as systemctl fails to talk to the system bus, and thus, the installation is marked as failed!

Oversights like this are exactly the reason behind my systemd dislike. We never wanted the systemd package, but another key package suddenly added it as a dependency one day...

Now to see if this is reported as a bug already, and if not, to report it myself...

(also, who checks for init by looking for the init's management utility?! Its like I checked if sysvinit is installed by checking if update-rc.d is installed!
And not like figuring out the system's init daemon is hard anyway! Just check /bin/init, or, better yet, check for process with pid 0!)

A very long rant.. but I'm looking to share some experiences, maybe a different perspective.. huge changes at the company.

So my company is starting our microservices journey (we have a 359 retail websites at this moment)

First question was: What to build first?

The first thing we had to do was to decide what we wanted to build as our first microservice. We went looking for a microservice that can be used read only, consumers could easily implement without overhauling production software and is isolated from other processes.

We’ve ended up with building a catalog service as our first microservice. That catalog service provides consumers of the microservice information of our catalog and its most essential information about items in the catalog.

By starting with building the catalog service the team could focus on building the microservice without any time pressure. The initial functionalities of the catalog service were being created to replace existing functionality which were working fine.

Because we choose such an isolated functionality we were able to introduce the new catalog service into production step by step. Instead of replacing the search functionality of the webshops using a big-bang approach, we choose A/B split testing to measure our changes and gradually increase the load of the microservice.

Next step: Choosing a datastore

The search engine that was in production when we started this project was making user of Solr. Due to the use of Lucene it was performing very well as a search engine, but from engineering perspective it lacked some functionalities. It came short if you wanted to run it in a cluster environment, configuring it was hard and not user friendly and last but not least, development of Solr seemed to be grinded to a halt.

Elasticsearch started entering the scene as a competitor for Solr and brought interesting features. Still using Lucene, which we were happy with, it was build with clustering in mind and being provided out of the box. Managing Elasticsearch was easy since there are REST APIs for configuration and as a fallback there are YAML configurations available.

We decided to use Elasticsearch since it provides us the strengths and capabilities of Lucene with the added joy of easy configuration, clustering and a lively community driving the project.

Even bigger challenge? Which programming language will we use

The team responsible for developing this first microservice consists out of a group web developers. So when looking for a programming language for the microservice, we went searching for a language close to their hearts and expertise. At that time a typical web developer at least had knowledge of PHP and Javascript.

What we’ve noticed during researching various languages is that almost all actions done by the catalog service will boil down to the following paradigm:

- Execute a HTTP call to fetch some JSON

- Transform JSON to a desired output

- Respond with the transformed JSON

Actions that easily can be done in a parallel and asynchronous manner and mainly consists out of transforming JSON from the source to a desired output. The programming language used for the catalog service should hold strong qualifications for those kind of actions.

Another thing to notice is that some functionalities that will be built using the catalog service will result into a high level of concurrent requests. For example the type-ahead functionality will trigger several requests to the catalog service per usage of a user.

To us, PHP and .NET at that time weren’t sufficient enough to us for building the catalog service based on the requirements we’ve set. Eventually we’ve decided to use Node.js which is better suited for the things we are looking for as described earlier. Node.js provides a non-blocking I/O model and being event driven helps us developing a high performance microservice.

The leap to start programming Node.js is relatively small since it basically is Javascript. A language that is familiar for the developers around that time. While Node.js is displaying some new concepts it is relatively easy for a developer to start using it.

The beauty of microservices and the isolation it provides, is that you can choose the best tool for that particular microservice. Not all microservices will be developed using Node.js and Elasticsearch. All kinds of combinations might arise and this is what makes the microservices architecture so flexible.

Even when Node.js or Elasticsearch turns out to be a bad choice for the catalog service it is relatively easy to switch that choice for magic ‘X’ or component ‘Z’. By focussing on creating a solid API the components that are driving that API don’t matter that much. It should do what you ask of it and when it is lacking you just replace it.

Many more headaches to come later this year ;)

Have you deployed ElasticSearch to production? If so, I got couple of questions for you.

How much complexity did ES add to the project overall from a developers perspective?

How much did this differ in price from other solutions you used? In production loads that is.

✌🏼

Implement a rest API for elasticsearch.
Follow the client's index's mapping.
Generate json document from Java pojos, given by the client.
Jsons don't match the schema mapping, one (at least) field, for geographic coordinates, is in another format.
Ask the client for explanation.
Client response, after 6 hours:
"We build it in this shape so you have to convert them to another format before posting into ES".
What the hell is wrong with you?!

!!rant
Elasticsearch! First time touching it and need to find out on my own how to build an index that allows a weighted multi field fuzzy search on four fields where two needs to be full ngram, one ngram on the words and one standard search + not index any other field. The documentation is horrible! Just realizing that this is what I need took me 2 days!

Pro tip by a Noobie: Whenever you use an open sourced software, and set it up using some tutorial, make sure you download the latest distribution.

Wasted 2 days fixing something while setting up KeyCloak, eventually downloaded the latest version and worked fine. There was a bug in KeyCloak apparently.

Happened the same 2 and a half years ago trying to write node scripts for ElasticSearch, using an older ES library -_-

Has anybody here used Solr or Elasticsearch for a big online shop? We’re implementing fact finder and are not happy and are wondering about solr and stuff like that.

I kinda want more emotional input from other devs so I thought I asked here :)

Thanks !

Is there some sort of Query Builder for ElasticSearch?

I have ELK setup and in Kibana can generate all the aggregation visualizations but now I want the data to be usable in a program so it can generate reports like who are our top users.

But the aggregation queries seem to be very verbose... not sure how anyone can generate or understand it by hand vs telling Kibana I want a chart with X and Y axes using these terms.

IDeally I'd like to have Kibana then tell me what's the actual JSON/Elastic query it used to generate that but can't seem to find something like that.

I had a discussion - no, it was more a lobotomy - with one of our "experts"

I was kinda confused, as he had several grafana tabs open and an query editor...

He explained to me that he debugs and optimizes his query based on the grafana data....

Elasticsearch cluster with several hundred, different indices, > 20 TB data

I explained to him the scrape interval of 5secs, that he cannot distinguish his query from other queries, that there is far too much of an interference... Let alone that a 5 sec scrape interval is a very loooong time.....

Nope. It makes perfect sense to him and he'll continue to work like this.

Wonderful weekend and day today. Spent the weekend getting ELK up and running, creating a backend and frontend in python for the data in elasticsearch, brushing up my node skills by rewriting some old grails services in node, and built my first app in Flutter. Spent today getting three Intel NUCs running docker, jenkins, and nexxus. Not too bad.

From last one week I m just sleeping for 2 hours trying to solve a problem in elasticsearch by writing plugin .. its always fun i never understand when it becomes 7am in the morning.

What do you guys think about deploying elastic search on App Engine Custom Runtime?

(Basically, an empty folder with an elastic search Dockerfile.)

I think it's a good idea: you can now deploy your code and storage application (Elastic search, Redis, etc) as services on your cluster.

You can use GCP magic to auto scale those services, you have so many good stuff that come with it.

And it's inside the same network as your services running in the same AppEngine project.

We were on track to provision Cassandra for logging and elasticsearch for business data store.
Now we are on track to provision mongodb and elasticsearch. That's right, two document stores.

Things go to hell when management think they are capable of taking engineering decisions.

This invite to an ElasticSearch webinar is epic:

webinars/proven-architectural-patterns-for-mature-elastic-stack-deployments?ultron=reference-architecture-webinar&blade=invite&hulk=email

You know how people rant about js frameworks; well the very same is true about nosql.

I thought let me broaden my horizon (pun intended) with a nosql db in my project.
So from Friday evening, I started off with ElasticSearch, which is pretty simple to get started, but apparently I need to understand it a lot better to use it as a primary data-store.
Then I stumble upon orient-db, was pretty exciting and learnt the apis/librarys but researching it a bit more to learn about the community; there is some bad-blood there.
Now I'm onto something called ArangoDB, think I'll stick with this; Any more time spent on this and I'll just give up on the project.

ElasticSearch vs. Solr

What do u prefer and why?

I did some of the front-end and whole backend. build and manage the SQL + elasticsearch database. After all of this, only 17 lines of mother fu**er code ruined my life. The client is asking for code. And.... And... Can't say anymore.

input {
file {
path => "/home/rsa-key-20200528/aslogger.log"
type => "java"
start_position => "beginning"
}
}

output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => ["localhost:9200"]
index => "aslogger"
}
}

Getting an ElasticSearch Developer training this week, anyone here been already? If so, what was it like?

How the fuck do you use and make a fields.yml for dynamic filebeat indexes?

Aka what if i don't want all the fields?