Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
Search - "elastic"
Privacy & security violations piss me off. Not to the point that I'll write on devRant about it, but to the point that coworkers get afraid from the bloodthirsty look in my eyes.
I know all startups proclaim this, but the one I work at is kind of industry-disrupting. Think Uber vs taxi drivers... so we have real, malicious enemies.
Yet there's still this mindset of "it won't happen to us" when it comes to data leaks or corporate spying.
Me: "I noticed we are tracking our end users without their consent, and store not just the color of their balls, but also their favorite soup flavor and how often they've cheated on their partner, as plain text in the system for every employee to read"
Various C-randomletter-Os: "Oh wow indubitably most serious indeed! Let's put 2 scrumbag masters on the issue, we will tackle this in a most agile manner! We shall use AI blockchains in the elastic cloud to encrypt those ball-colors!"
NO WHAT I MEANT WAS WHY THE FUCK DO WE EVEN STORE THAT INFORMATION. IT DOES IN NO WAY RELATE TO OUR BUSINESS!
"No reason, just future requirements for our data scientists"
I'M GRABBING A HARDDRIVE SHREDDER, THE DB SERVER GOES FIRST AND YOUR PENIS RIGHT AFTER THAT!
(if it's unclear, ball color was an optimistic euphemism for what boiled down to an analytics value which might as well have been "nigger: yes/no")13
C'mon people! Spread the word! "The cloud" is not "just someone elses computer", it's a completely different way to compute!
I'm so tired of the oversimplifications done trying to explain the consept. The massive amount of work, sweat and tears put into the orchestration, automation and abstraction layers to deliver truly elastic, scalable and self healing infrastructure, applications and services deserves a fuckload more respect than "just someone elses computer"!
Hosting and time-sharing have been with us almost as long as we have had computers (mainframes etc), but dismissing the effort of thousands upon thousands of devs and ops people to make systems robust and automated enough to literally being able to throw a wrench in the engine any time during production and not have the systems suffer is fucking insane!
The whole reason the term "cloud" is so fitting is not just because it was coined from the cloud-shape used in technical and non-technical drawings and illustrations symbolising the internet, but also because of the illusion of magic it gives the end-user not being able to see "whats inside the music box".19
So today was the worst day of my whole (just started) career.
We have a huge client like 700k users. Two weeks ago we migrated all their services to our aws infrastructure. I basically did most of the work because I'm the most skilled in it (not sure anymore).
Today I discovered:
- Mail cron was configured the wrong way so 3000 emails where waiting to be sent.
- The elastic search service wasn't yet whitelisted so didn't work for two weeks.
- The cron which syncs data between production db en testing db only partly worked.
Just fucking end me. Makes me wonder what other things are broken. I still have a lot to learn... And I might have fucked their trust in me for a bit.13
A project I'm working on uses Elastic for internal monitoring and logs. The customer asked to access those logs - not something we'd normally do, but it's isolated from other things we use and there's no critical data there, so what the heck, let them have it.
Ever since, we're getting tons of questions like "There are tons of [insert random info message] all the time, do you have any plans to resolve them?" and it gets to the point where I'm just about ready to scream back "NO, SUZAN, BOOKING NOT COMPLETED MANS THE USER F###ING CANCELLED IT, IT'S NOT SOMETHING I CAN FIX IN THE CODE"
Edit: the customer's name isn't actually Suzan4
Been developing with AWS EC2 since 2015. Have deployed multiple apps in my Ubuntu servers. And have deployed Dockerized apps into Elastic Beanstalk. And used S3 to store files.
But just because I don't know AWS Lambda, the recruiter thought I'm not qualified enough for the job.
Elasticsearch queries are FUCKING ugly
Elasticsearch documentation is FUCKING bare-minimum.
Kibana only shows data when it FUCKING feels like it.
Elastic stack is FUCKING annoying me12
Yay! Professional ranter now on phone and together with GCP, devops amsterdam, elastic and others!1
Using mongodb for one product
A colleague as experimenting with elastic search (I think it was).
It installed a proxy around the collection to get all events for the external search storage.
Worked well, but it was just a test so once done we removed it
But thats where it got scary.
When we removed the proxy through the search dashboard it dropped the underlying collection of live data!!!
A collection it did not create.
Hows that for bad UI.
Always experiment on a separate db server.
Simulating a couple million elastic search searches and aggregations in 2h instead of 3 months ... :)3
My boss did not care about making things secure in our early development stage, even though I told him several times.
After 1 day our elastic search cluster was filled with random crappy data.
Fix: Apply security schemes provided by AWS1
Today at work I accidentally redeployed our ELK instance without taking a backup of all of Kibana's saved objects...
I didn't realize Elastic stores all indexes including the Kibana's in it's app folder by default.
Tomorrow will be fun.... I can't decide what to do first... Recreating all the charts and tables from memory... Or fixing the deployment script to change the data dir path...2
I'm so sick of microservice architecture... in theory it was going to make scaling elastic and deployment easier. In reality it seems to slow productivity to a 🐌 pace.
Anyone have any brilliant suggestions on how to herd these cats in production?9
Note to self: Pointing your tests at a non disposable DB will cause very very bad things to happen. No idea what the flying fuck I was thinking - but praise to the data gods it wasn't a production elastic!
Am I the only one who just loves when I do "curl 127.0.0.1:9200/_template" and get back entire 32" screenful of compressed JSON?
If you ever want to know how NOT to create documentation, try installing Elastic Search for a server.
Fuck this Apache Server on this elastic beanstalk!!! Nginx is so much better at configuration and ease of use!!!4
How much do you earn for your skill set in your country vs your cost of living?
See how much I & others earn.
Recently I became aware of just how massive the gap in developers earnings are between countries. I'd love to calculate a fixed score for income vs cost of living.
I know this stuff is sensitive to some so if you prefer just post your score (avg income p/m after tax / cost of living).
I'm not shy so I'll go first:
Normal Rate (Long term): $23
Consulting / Short term: $30-$74
Pen Test: $1500 once off.
Pen Test Fixes: consulting rate.
Simple work/websites: min $400+
Family & Friends: Dev friends are usually free (when mutually beneficial). Family and others can fuck off, even if they can pay (I pass their info to dev friends with fair warning).
Experience: 9 years
Country: South Africa
Developer rareness in country: Very Rare (+-90 job openings per job seeker).
Middle class wage in country: $1550 p/m (can afford a new car, decent apartment & some luxuries like beer/eating out).
Employment type: Permanent though I can and do freelance occasionally.
Client Locality: Mostly local.
Developer Type: Web Developer (True web dev - I do anything web related from custom HTTP servers to sockets, services, advanced browser api's, apps & more).
STACKS / SKILLSETS
I'M PROFICIENT IN:
I DABBLE WITH:
ASP.net, C++, ruby, GO, nginx, tesseract
application architecture, automation, integrations, db's, real time data, advanced browser apps/extensions (webRTC, canvas etc).
Avg income p/m after tax: $2250
Cost of living (car+rent+food): $1200
*Note: For integrity when calculating my cost of living I excluded debt repayments and only kept my necessities which are transport, food & shelter.
I really hope you guy's post your results, it would be great to get an idea of which is really the worst / best country to be a developer in.20
Need Advice + Rant
I am an Android Developer, pursuing an Internship, which i thought would be good for my career. But I am being assigned the task to build search feature for the App using Elastic Search. I intially was halpy to work on Search since it had to be Algolia. I am hating the work now because I am getting so stuck with Elastic and there have been other factors which also have decreased my productivity, but I am being quite inefficient. Now the deadlines are coming closer and if I dont give output I will be laid off. I am thinking about quitting myself because now I feel extremely demoralized and demotivated to work because we first decided to work on Algolia and it was all ready before we thought of shifting to heroku and now on AWS. What do the experienced once suggest? It's not that its impossible to do, now i just have to write queries in Java, again I am stuck and not really looking forward to since I was given the deadline today, for 2 days later.
The only issue is, I may have to return the new phone (OnePlus 3T) which I bought planning to later return the money to someone through my stipend.23
So I was setting up ELK (Elasticsearch, Logstash and Kibana) all in one EC2 on AWS today for demo purposes. I had everything prepared. Elastic IP, correct security group rules, etc.
I figured I would just do quick test before writing filters and templates if I can access Kibana. So I started service for it and tried to open it with Chrome.
Checked config file. Compared it to documentation. Seemed good but changed some things just for sake of change. Restarted service.
Reverted changes I've made in config. Restarted service. Curl on localhost. It work... OK. 😐
It took me half an hour but finally I figured it out after I took my phone and opened it from there. It was working from the beginning. Stupid company network was for some reason blocking this connection. Fuck! 😡And I was restarting that poor service like crazy trying to fix something that wasn't broken.
FUCK YOU AMAZON! Stop telling me my configuration yaml file contains a tab... THE FUCKING FILE DOES NOT EVEN EXIST!
In a time where a web dev is expected to know, well.. everything... Backend -JAVA, python, nodejs and C++ would be great.
Front- angular, react, other 10 libs
DBs -sql, mongo, redis, elastic, kafka, rebbitmq
Also be devops on the side with AWS and docker kubernetis and more stuff
How the f is that possible?
In my real job for the last couple of years and different companies, I usually use 1 language/framework & 1 main DB.. and although it's possible in some companies, but in mine, ppl dont get access to AWS etc..
So let's say there's me.. a server side dev for years.
So I decide to be better and learn Golang.. cool lang, never needed in my job, after few days of not using it I forgot all I learned and that was it.
Then I realized I gotta know some frontend cause everyone want a fullstack ninja nowadays.. so I tried Vuejs.. it was amazing .. never got to use it at work, cause i was a backend, and we didnt use frameworks on our products back then..
Then I decided to learned nodejs, because this is the coolest thing ever.. hated it, but whatever... Never got to use it at work, cause everything was written in other lang which the whole team knew... Forgot the little i knew.
Then I decided, its time to see what Angular is, cause everyone started using it... similar idea to vuejs which i barely remembered, but wow it's a lot of code to remember, or I'll have to google everything.. so I went over it, but can't say i even learned it.
Now Im trying to move on to python, which, I really am learning in depth.. however, since I dont have real experience with it, no one gives me a shot at being a python dev, so again i feel like I'm trying to memorize syntax and wasting my time..
Tired of seeing React in all job ads, i decided to have a look what's that all about.. and whadoyaknow... It's fucking the same idea as vue/angular with again different syntax..
THIS IS CRAZY!
in how many syntaxes do i need to know how to make a fucking crud api, and a page with same fucking post form, TO BE A GOOD PROGRAMMER?!?6
So, since almost a week I was trying to get familiar with Algolia which was to be used as API for the search feature in our App. But now, we are going with Elastic Search.
Who gives the task of building an Elastic Search feature to be deployed using Heroku on a Firebase database to an intern? I have never been so stuck, so lost, so confused, so damned, so fused, so annoyed, so pissed off, so irritated, so "hate my life", so "hate programming, I should start selling mangoes", so fml ever before in my whole life. :(7
I hate the elasticsearch backup api.
From beginning to end it's an painful experience.
I try to explain it, but I don't think I will be able to cover it all.
The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)
The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.
Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots
Workaround... Create many repositories as each new repository forces an full backup.........
The second thing: ES scales. Many nodes / es instances form a cluster.
Usually backup APIs incorporate these in their design. ES does not.
If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.
It might sound not so bad with 12 nodes and one index...
But it get's pretty bad with 100s of indexes and several dozen nodes...
And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.
So traffic shaping must be manually added. Yay...
The last thing is the API itself.
It's a... very fragile thing.
Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.
Documentation != API != Reality.
Especially the fault handling left me more than once speechless...
gives you a state PARTIAL
gives you a state SUCCESS
Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.
And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).
So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...
Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.
So much joy and fun, isn't it?
Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.
PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.
I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.2
This is a part rant-part question.
So a little backstory first:
I work in a small company (5 including me) which is mostly into consultation (we have many tech partners where we either resell their products or if there is a requirement from one of our clients, we get our partners to develop it for them and fulfill the client requirements) so as you can see there is a lot of external dependencies. I act as a one-hat-fits-all tech guy, handling the company websites, social media channels, technical documentation, tech support, quicks POCs (so anything to do with anything technical, I handle them). I am a bit fed up now, since the CEO expects me to do some absurd shit (and sometimes micro manages me, like WTF I am the only one who works there with 100% commitment) and expects me to deliver them by yesterday.
So anyway long story short, our CEO finally had the brains to understand that we should start having our own product (which i had been subtly suggesting him to do for a while now!).
Now he came up with a fairly workable concept that would have good market reach (i atleast give him credits for that) and he wanted me to suggest the best way to move forward (from a both business and technical point of view). The concept is to have an auction-based platform for users to buy everyday products.
I suggested we build a web app as opposed to a mobile one (which is obvious, since i didnt want to develop a seperate website and a mobile app, and anyway just because we can doesnt mean we have to make a mobile app for everything), and recommended the Node/react based JS tech stack to build it.
At first he wanted me to single handedly build the whole platform within a month, I almost flipped (but me being me) then somehow calmed down and finally was able to explain him how complicated it was to single-handedly build a platform of such complexity (especially given my limited experience; did I mention that this is my first job and I am still in college, yeah!!) and convinced him to get an experienced back-end dev and another dev to help me with it.
Now comes the problem, I was to prepare a scope document outlining all the business and technical requirements of the project along with a tentative cost, which was fairly straightforward. I am currently stuck at deciding the server requirements and the system architecture for the proposed solution (I am thinking of either going with AWS - which looks a bit complicated to setup - or go with either Digital Ocean or Heroku):
I have assumed that at peak times we would have around 500-1000 users concurrently
And a daily userbase of 1000 users (atleast for the first few months of the platform running)
What would be the best way forward guys?
I did some extensive (i mean i read through some medium blogs! and aws documentation) research and put together the following specs (if we are going through AWS):
One AWS t3.medium ec2 instance for the node server (two if we want High Availability by coupling with the AWS load balancer and Elastic Beanstalk)
The db.t3.small postgres database
The S3 Storage bucket (100gb) for the React Front end hosting
AWS SNS for email/sms OTP and notification
And AWS CloudMonitor for logging amd monitoring.
Am I speculating the requirements properly, where have I missed??
Can u guys suggest what is the best specification for such a requirement (how do you guys decide what plan to go with)?
Any suggestions, corrections, advices are welcome3
Need some advice here.
So hello everyone! I recently moved abroad for work, for the sake of the experience and the excitement of learning how developers in Latin America tackle specific problems. To my surprise, the dev team is actually composed solely of Europeans and Americans.
I work for a relatively new startup with an ambitious goal. I love the drive everyone has, but my major gripe is with my team lead. He's adverse to any change, and any and all proposals made to improve quality of throughput are shot down in flames. Our stack is a horrendous mess patched together with band-aids, nothing is documented, there are NO unit tests for our backend and the same goes for our frontend. The team has been working on a database/application migration for about a month now, which I find ridiculous because the entire situation could have been avoided by following very rudimentary DevOps practices (which I'm shunned for mentioning). I should also add that for whatever reason containerization and microservices are also taboo, which I find hillarious because of our currently convoluted setup with elastic beanstalk and the the constant complaints between our development environment and production environments differing too much.
I've been tasked with managing a Wordpress site for the past 3 weeks, hardly what I would consider exciting. I've written 6 pages in the past two weeks so our marketing team can move off of squarespace to save some money and allow us more control. Due to the shit show that is our "custom theme" I had to write these pages in a manner that completely disregard existing style rules by disabling them entirely on these pages. Now, ironically they would like to change the blog's base theme but this would invertedly cause other pages created before I arrived to simply not work, which means I would have to rewrite them.
Before I took the role of writing an entire theme from scratch and updating these existing pages to work adequately, I proposed moving to a headless wordpress setup. In which case we could share assets in a much more streamline manner between our application and wordpress site and unify our styles. I was shot down almost immediately. Due to a grave misunderstanding of how wordpress works, no one else on the team seems to understand just how easy it is to fetch data from wordpress's api.
In any event, I also had a tech meeting today with developers from partner companies and realized no one knew what the fuck they were talking about. The greater majority of these self proclaimed senior developers are actually considered junior developers in the United States. I actually recoiled at the thought that I may have made a great mistake leaving the United States to look a great tech gig.
I mean no disrespect to Latin America, or any European countries, I've met some really incredible developers from Russia, the Ukraine, Italy, etc. in the past and I'm certainly not trying to make any blanket statements. I just want to know what everyone thinks, if I should maybe move back to the states and header over to the bay/NY. I'm from the greater Boston area, where some really great stuff is going on but I guess I also wanted a change of scenery.2
I've been putting a web application together that I'm looking to have published on AWS. Not having too much experience with AWS, I am looking to hire a contractor. I've had a number of quotes from different AWS admin's ranging from $40 an hour to $200 an hour, from 1-days worth of work to 2-months worth of work!
I'm not really sure what to make of it or to whom to trust. I believe they’re using my ignorance to overcharge me. I've listed my requirements below, could you guys use your professional experiences to let me know what you think is reasonable charge and where best I could find someone to help me.
My application is a US shopping website where people can set up an online shop and upload their products and maintain an inventory of the items.
This is what I’m looking for setup and configuration with the following two areas:
1) AWS SYSTEMS…
* AIM - Set up my server admin users.
* EC2 - Web Hosting.
* RDS - Fast DB.
* SES - To send emails.
* S3 Buckets - Uploaded image hosting.
Route 53 - I don’t know but someone said I should have this.
* Elastic Load Balancing - For, well, load balancing.
* A script that would back up the database once a day and save it to a private S3 Bucket.
* A script that will run once a day that calls an internal API, and POST a query to it.
* A script that runs once every 90 days, to refresh the SSL using ZeroSSL.com
Is there anything that I've missed such as security systems, firewalls, auto scaling and CDNs?
The quotes that I've received arranged from $320 to $64,000. I know I am being abused because of my ignorance. I would never overcharge someone because the customer doesn't know the efforts of the work. I hope someone here can help to understand the efforts needed and can tell me the true cost.
To be a Java (or other business popular language) developer
* Java 6, 8 and features up to 14
* SQL + nosql
* Logging eg log4j2,
* Searching eg elastic stack
* Framework (at least 1, but hey, knowing 1 is lame..)
* Networking or at least base http knowledge
* Tomcat, jboss or other shit
* Aws, heroku, GCE or other SAAS/paas
* Rest, RPC, soap
* Business Hello World example
* Hexagonal Architecture
* 12 app factor
* Security, oauth2
* Eureka or consul as service Discovery
* Config server
* Hazel cast
* Endless story ...
Then we can start hello word app2
IT always remove elastic search package and i can't migrate my database because of this package required to be run. I always reinstall this to make it run.
Sometimes vagrant itself don't run perfectly in windows.
I think i will move to UBUNTU to make it run smooth.
TODAY you made my task stuck.2
The whole of the UK pretty much in standstill for a football match.
... and I'm here trying to get my fucking app to deploy into Glassfish 5 on AWS Elastic Beanstalk.2
Sat here trying to decide and finalise my Dev process for Wordpress!
Roots.io clean, good code, deployment to staging and production through git
Vagrant then just push to live (which one?)
Docker then try and figure S*** out
Then decide where to deploy:
Digital Ocean or AWS Elastic with load balancers and S***!
So I just found out I have to use amazon (instead of Heroku) for hosting a service I'm supposed to build. The environment just f**king refuses to bild and these are my logs. fml...
- Finish "Introduction to algorithms"
- Learn some genetic algorithms
- Get my hands dirty on reinforcement learning
- Learn more about data streaming application (My currently app is still using plain stupid REST to transport image). I don't know, maybe Kafka and RabbitMQ.
- Learn to implement some distributed system prototypes to get fitter at this topic. There must be more than REST for communicating between components.
- Implementing a searching module for my app with elastic search.
- Employ redis at sometime for background tasks.
- Get my handy dirty on some operating system concepts (Interprocess Communication, I am looking at you)
- Take a look at Assembly (I dont want to do much with Assembly, maybe just want to implement one or two programs to know how things work)
- Learn a bit of parallel computing with CUDA to know what the hell Tensorflow is doing with my graphic card.
- Maybe finishing my first research paper
- Pass my electrical engineering exam (I suck at EE)1
Finally started utilizing this quarantine time and started a new project.
Name - Hermes
Link - https://github.com/gauravat16/...
About - Send Cloud message notifications (like FCM) to your users.
Features I am planning -
1. Send notifications to users based on any specification you want. (eg - users on app version 1.2 and using OS version 9.0 or 8.0 in region India)
2. Search on previous requests and responses.
3. Draw trends on the responses and further actions by the user.
Current tech stack -
2. Java 1.8
5. Elastic (Planned)18
Fucking AWS Elastic Beanstalk took a week of work to get fucking mounts set up. They invented their own version of docker compose that is missing half the features so I had to work my way from hacking their scripts to include options they don't support, to restarting the whole fucking docker service on every fucking deployment and now the shit finally works. How can most StackOverflow answers just say restart docker, this shit is not ok! I fucking hate sysadmin work. I want to code :(
Manual EC2 instances + Elastic load balancer or Elastic beanstalk for a PHP 7 application? I might have some cron jobs to be run too...
What's a good book to learn ELK hands-on?
I have an instance set up and working but want to need some more advanced features? like mapping, index templates, querying remotely for aggregations.
The Elastic docs feel very high level and maybe assumes u can read their minds... A lot of snippets I just go "uhm.. where do I put this in the file, which file?"4
What do you guys think about deploying elastic search on App Engine Custom Runtime?
(Basically, an empty folder with an elastic search Dockerfile.)
I think it's a good idea: you can now deploy your code and storage application (Elastic search, Redis, etc) as services on your cluster.
You can use GCP magic to auto scale those services, you have so many good stuff that come with it.
And it's inside the same network as your services running in the same AppEngine project.1
This invite to an ElasticSearch webinar is epic:
Is there some sort of Query Builder for ElasticSearch?
I have ELK setup and in Kibana can generate all the aggregation visualizations but now I want the data to be usable in a program so it can generate reports like who are our top users.
But the aggregation queries seem to be very verbose... not sure how anyone can generate or understand it by hand vs telling Kibana I want a chart with X and Y axes using these terms.
IDeally I'd like to have Kibana then tell me what's the actual JSON/Elastic query it used to generate that but can't seem to find something like that.1
For six months, I have tried to put geodjango on AWS elastic beanstalk and failed. Anyone with a working solution?2
used bootstrap, and the found a prety elastic side menu animation, turned out it can'twork with the best css fw.. fml
Where can find an all in one guide to set up/configure a ELK environment manually from the tars (not yum, apt-get, docker image)?
I am following the component docs from Elastic but not sure how each component integrates with each other or how to set the mongo connection (DB is not local)5
Hi fellow devRanters, I need some advice on how to detect web traffic coming from bad/malicious bots and block them.
I have ELK (Elastic) stack set up to capture the logs from the sites, I have already blocked the ones that are obviously bad (bad user-agent, IP addresses known for spamming etc). I know you can tell by looking at how fast/frequently they crawl the site but how would I know if I block the one that's causing the malicious and non-human traffic? I am not sure if I should block access from other countries because I think the bots are from local.
I am lost, I don't know what else I can do - I can't use rate limiting on the sites and I can't sign up for a paid service cause management wants everything with the price of peanuts.
Someone asked why I can't just read through the logs (from several mid-large scale websites) and pick out the baddies.
*facepalm* Here's the gigabytes log files.9
Django project in a docker container in an ec2 instance of my aws vpc... i did use elastic beanstalk though, but needed to let my media files be uploaded to S3... couldn't figure it out yet...
Sophomore year starting soon so I'm looking for new project (s) to complete in parallel with the studies.
Some are more design-y and some more backend-y but I recently started getting better at designing so :)
1) Learn some fragment shader stuff. I've always been messing around with graphics and have a game on steam, so I think that's a good idea to be paired with signal processing.
2) Reactive web services. Preferably with spring-boot or vert.x but
3) I would also like to dive into golang (and make some reactive thing with it)
4) WebAssembly seems nice... But I got some concerns
5) exercise making wireframes -> CSS (with some js)
6) I've never really done any real backed work with nodejs, except serving and aot compiling js, or doing gulp tasks
7) Implementing a whole project, or a fraction of it as serverless on aws
* I'm definitely going to use a couple very simple services to make a docker swarm with load balancing, etc, just because I know how everything works but got no practical knowledge
8) Design an esports jersey for the university department I'm in (shouldn't take long)
So what do you guys think? Recommendations are welcome :)
P.S. last year in review:
> A webapp running on a raspberry pi powering a reflex testing game on gpio (java/spring-boot , codename: buttonmasher)
> small Elastic search cluster to monitor some random university servers through kibana dashboards
> laser tracking on wall of *any* colour and variable light conditions via a webcam (opencv) , controlling the mouse pointer, whether you run it against a projector or any wall
> Various random Photoshop stuff
Any good recommendations on how to gather user metrics/instrumentation and visualize data?
The program is a WPF application with not internet connection so logging to file and get file is probably the only solution.
I've played a little with Serilog to file and trying to import the log into elastic search and visualize data with Kibana.
How the fuck do you use and make a fields.yml for dynamic filebeat indexes?
Aka what if i don't want all the fields?