migrated everything to elastic search, shitty learning curve and apparently shittier performance. now i dream of going back

Welp, here goes my perfs upgrade for next week.

Is it really so bad ?

ES is as usefull, if you put in the effort to make it usefull.

It is pretty easy to just dump everything into an External ES cluster, and then never look at the logs again.... You need to put in time and effort to get somwthing usefull out of it.

It's times like these when I realize I need to go back and I'm glad I'm using Git.

ES is pretty performant.

But it has definitely a steep learning curve and you need a lot of knowledge to make it work.

Plus a lot of patience, architecture design and benchmarking.

@NoToJavaScript there are not as many helpful youtube tutorials, but if you somehow learn a lot about it, try several approaches, you could probably get it to be perfromant.

im one month in, watched udemy courses, read every elastic discussion about my issue, asked ppl on reddit, still wasnt able to get it as fast i want it to be but many people said they got the results i want

@sleek what did you use before?

@eval Algolia. but in the end it became too expensive which is why we wanna move to elastic.

Algolia tho was returning search results within 40ms and I need to get elastic to do the same to make a case for the move

@sleek if you have *detailed* questions, I might answer them.

@IntrusionCM well im trying to implement user search on elastic. I would search using a user's name. I have 60 million users.

my cluster has 3 dedicated master nodes and 4 data nodes (2vcpu 16gb ram, 112gb nvmeSSD)

i created my user index with 4 primary shards and 1 replica. I also tried having it on 1 primary and 3 replicas. both combinations gave me slow results 70ms to 600ms. the bigger the search query the longer it takes.

my 60m user documents (they just have a name and a picture set to index=false) take up only 3gb of user data on primaries.

so what am i doing wrong? do i need more nodes ? bigger nodes? more shards ? less shards ?

algolia seems to be responding within 30-40ms regardless of query length and in hoping to achieve the same speed in elastic

thanks in advance for ur help :/

@sleek

I'll just write down what comes immediately to my mind...

The usual concept of ES is:

- master node

- search node

- data node

Master Node: Don't use them for loadbalancing or anything else. They need lil hardware, especially in small clusters, 4 GiB is more than enough in small use cases ( 2 GiB JVM, rest system ). They just exist to be used by the data / search nodes for cluster tasks.

Search Node: Have one search node, it's the one node who should be used for loadbalancing.

It "parses" the incoming HTTP request, splits it into ES operations and forwards it via ES TCP protocol to the data node.

Data Node: Again, should never be used for loadbalancing. At least 2 are required.

The search / data nodes are usually the beefy ones - max. half of the RAM should go to the JVM, rest for System and Lucene FS cache.

In your case, I would go for 8 GiB JVM, rest Lucene / OS.

--

Note: I'm uncertain if they finally provided good systemd units by default, the overrides set what whould previously accomplished by ulimits.

--

systemctl edit elasticsearch.service (on each node):

Content:

[Service]

# disable private temp protection

PrivateTmp=false

# Specifies the maximum file descriptor number that can be opened by this process

LimitNOFILE=65536

# Specifies the maximum number of processes

LimitNPROC=8192

# Specifies the maximum size of virtual memory

LimitAS=infinity

# Specifies the maximum file size

LimitFSIZE=infinity

# Limits memlock

LimitMEMLOCK=infinity

... or you create a directory:

/etc/systemd/system/elasticsearch.service.d

with a file ending in ".conf".

(command does the same, names the file override.conf)

Then the basic stuff should be done:

JVM / SystemD / sysctl.

I'd recommend checking via Grafana / Prometheus or "raw" parsing of node health API

(https://elastic.co/guide/en/...) what goes wrong.

Kibana has a Query Debugger - which can provide excellent insight in what the query does and what not.

Remaining hints:

- check the index refresh time. if you build an index in one step, disable index refreshing via index settings

- if you do insert regularly in the index, make sure that index merging works as intended

---> Note: make sure you do not insert "document by document", the more often you trigger index merging, the less stable your query search time will be (due to interference of refreshing / merging)

- check the analyzer settings and make sure you provide a good index template, there is a lot to tweak

@IntrusionCM Hmmm, we index usuer upon them changing profile info like names or upon registration. so we constantly send index calls to elastic. i guess we need to accumulate them to be done once or twice a day?

also im using aws managed elastic search so i was given the option to assign master and data nodes. they ddnt ask me about search nodes, i hope i wont have to configure a cluster myself tho

i still wonder tho why longer queries take more time than shorter ones, could fuzziness be that time consuming?

@sleek ah okay.

I'm not having any info about AWS.

Regarding longer vs shorter... What do you mean exactly. What does longer mean?

Accumulation of updates to batches would be good.

You can manually refresh an index.

Thus you could disable index refresh via index settings, do an bulk insert (e.g. once every 5 mins), flush changes, refresh.

That would eliminate ES constantly checking in background if the index has to be refreshed.

@IntrusionCM ah i see, manually refreshing after bulk inserts one every X interval seems interesting ill definitely try that thank you!!

regarding longer i mean if a user searches for "john doe smith" with fuzziness set to auto takes longer than if he searches for "john" for instance. is my observation logical or u think its a coincidence?

@sleek

Need more information. Fuzziness is a specific term:

https://elastic.co/guide/en/...

Do you use fuzzy query **AND** do you have a wildcard family keyword field?

https://elastic.co/guide/en/...

I always recommend - to catch typical brainfarts during implementation of index templates - to set search.allow_expensive_queries to false.

Queries will be rejected, which is the best, as you then have a failure / exception in code, directly leading you to where the query was executed via stack trace.

https://elastic.co/guide/en/...

You should read regarding text / keyword fields this blog entry.

I highly recommend it as it gives good basic knowledge:

https://elastic.co/de/blog/...

@IntrusionCM I tried disabling auto refresh and make jobs to bulk index data and trigger refresh after its done.

i got it to 30ms :'). Thank you. Thank you so much

@sleek 👍🥞🍻