Often I hear that one should block spam email based on content match rather than IP match. Sometimes even that blocking Chinese ranges in particular is prejudiced and racist. Allow me to debunk that after I've been looking at traffic on port 25 with tcpdump for several weeks now, and got rid of most of my incoming spam too.

There are these spamhausen that communicate with my mail server as much as every minute.
- biz-smtp.com
- mailing-expert.com
- smtp-shop.com

All of them are Chinese. They make up - rough guess - around 90% of the traffic that hits my edge nodes, if not more.

The network ranges I've blocked are apparently as follows:
- 193.106.175.0/24 (Russia)
- 49.64.0.0/11 (China)
- 181.39.88.172 (Ecuador)
- 188.130.160.216 (Russia)
- 106.75.144.0/20 (China)
- 183.227.0.0/16 (China)
- 106.75.32.0/19 (China)
.. apparently I blocked that one twice, heh
- 116.16.0.0/12 (China)
- 123.58.160.0/19 (China)

It's not all China but holy hell, a lot of spam sure comes from there, given how Golden Shield supposedly blocks internet access to the Chinese citizens. A friend of mine who lives in China (how he got past the firewall is beyond me, and he won't tell me either) told me that while incoming information is "regulated", they don't give half a shit about outgoing traffic to foreign countries. Hence all those shitty filter bag suppliers and whatnot. The Chinese government doesn't care.

So what is the alternative like, that would block based on content? Well there are a few solutions out there, namely SpamAssassin, ClamAV and Amavis among others. The problem is that they're all very memory intensive (especially compared to e.g. Postfix and Dovecot themselves) and that they must scan every email, and keep up with evasion techniques (such as putting the content in an image, or using characters from different character sets t̾h̾a̾t̾ ̾l̾o̾o̾k̾ ̾s̾i̾m̾i̾l̾a̾r̾).

But the thing is, all of that traffic comes from a certain few offending IP ranges, and an iptables rule that covers a whole range is very cheap. China (or any country for that matter) has too many IP ranges to block all of them. But the certain few offending IP ranges? I'll take a cheap IP-based filter over expensive content-based filters any day. And I don't want to be shamed for that.

!dev
This boring story with stupid ending started on Monday with me going out to buy some food and cook something delicious, day like always until my mind went nuts.

I work from home and cook my meals by myself cause I love cooking.
To buy ingredients I go shopping couple times a week always making the same steps, doing this for over a year now and by this time everything was automatic so I could think about work problems and solutions.

I start usually by getting up from my desk around noon, not many people doing shopping at that time and I can proceed quick.
Algorithm is like this: go to kitchen and look at the fridge, go out, wait for traffic lights, take tram, ride two stops, wait for the traffic lights again, go to supermarket, do shopping and finally go back the same way. Boooring.

When I get out from tram that day l looked at traffic lights to go green, as always and that’s the place where everything started to go bad.

So I was waiting there doing nothing and then stupid idea got me.
I figured out I can stop looking at light to make this day different and look ahead.
Then simply start walking when people from other side start walking.

It worked smoothly on those lights and I was happy I can do things differently from now on. I proceed with this idea on the way back and motherfuckers started walking on red. Twice !!!!
Almost died.

Since then three times some car was driving on green near me in those places and people started walking on red.

It got me worried about world determinism instantly. I might increased some entropy to much and some world developer changed some line of code while I was shopping and from that time death is passing by me.

Now it got me to the point where the more I follow this way the more I am worried about my life. Started thinking about ordering ingredients online.

So if you read this you know that I know your plan and I will be changing supermarkets and paths to it randomly starting from next week.

Or not I hope nobody hacked my mind and only thing that read and write to it is my consciousness.

I feel relief now.

One of our integration solutions (via Webservices) had some issues. I had to switch on http logging to see what might be the issue.

On average, those logs are around 20MB when there is a bunch of traffic. But the solution brought a heap pf traffic through, those logs shot up to 1GB in size.

Had to delete the logs, since they took a million years to open, and told our vendors that the logs are not showing us anything 😅 I told no lies

Just posted this in another thread, but i think you'll all like it too:

I once had a dev who was allowing his site elements to be embedded everywhere in the world (intentional) and it was vulnerable to clickjacking (not intentional). I told him to restrict frame origin and then implement a whitelist.

My man comes back a month later with this issue of someone in google sites not being able to embed the element. GOOGLE FUCKING SITES!!!!! I didnt even know that shit existed! So natually i go through all the extremely in depth and nuanced explanations first: we start looking at web traffic logs and find out that its not the google site name thats trying to access the element, but one of google's web crawler-type things. Whatever. Whitelist that url. Nothing.

Another weird thing was the way that google sites referenced the iframe was a copy of it stored in a google subsite???? Something like "googleusercontent.com" instead of the actual site we were referencing. Whatever. Whitelisted it. Nothing.

We even looked at other solutions like opening the whitelist completely for a span of time to test to see if we could get it to work without the whitelist, as the dev was convinced that the whitelist was the issue. It STILL didnt work!

Because of this development i got more frustrated because this wasnt tested beforehand, and finally asked the question: do other web template sites have this issue like squarespace or wix?

Nope. Just google sites.

We concluded its not an issue with the whitelist, but merely an issue with either google sites or the way the webapp is designed, but considering it works on LITERALLY ANYTHING ELSE i am unsure that the latter is the answer.

I hate the elasticsearch backup api.

From beginning to end it's an painful experience.

I try to explain it, but I don't think I will be able to cover it all.

The core concept is:
- repository (storage for snapshots)
- snapshots (actual backup)

The first design flaw is that every backup in an repository is incremental. ES creates an incremental filesystem tree.

Some reasons why this is a bad idea:
- deletion of (older) backups is slow, as newer backups need to be checked for integrity
- you simply have to trust ES that it does the right thing (given the bugs it has... It seems like a very bad idea TM)
- you have no possibility of verification of snapshots

Workaround... Create many repositories as each new repository forces an full backup.........

The second thing: ES scales. Many nodes / es instances form a cluster.

Usually backup APIs incorporate these in their design. ES does not.

If an index spans 12 nodes and u use an network storage, yes: a maximum of 12 nodes will open an eg NFS connection and start backuping.

It might sound not so bad with 12 nodes and one index...

But it get's pretty bad with 100s of indexes and several dozen nodes...

And there is no real limiting in ES. You can plug a few holes, but all in all, when you don't plan carefully your backups, you'll get a pretty f*cked up network congestion.

So traffic shaping must be manually added. Yay...

The last thing is the API itself.

It's a... very fragile thing.

Especially in older ES releases, the documentation is like handing you a flex instead of toilet paper for a wipe.

Documentation != API != Reality.

Especially the fault handling left me more than once speechless...

Eg:
/_snapshot/storage/backup
gives you a state PARTIAL
/_snapshot/storage/backup/_status
gives you a state SUCCESS

Why? The first one is blocking and refers to the backup status itself. The second one shouldn't be blocking and refers to the backup operation.

And yes. The backup operation state is SUCCESS, while the backup state might be PARTIAL (hence no full backup was made, there were errors).

So we have now an additional API that we query that then wraps the API of elasticsearch. With all these shiny scary workarounds like polling, since some APIs are blocking which might lead to a gateway timeout...

Gateway timeout? Yes. Since some operations can run a LONG (multiple hours) time and you don't want to have a ton of open connections hogging resources... You let the loadbalancer kill it. Most operations simply run in ES in the background, while the connection was killed.

So much joy and fun, isn't it?

Now add the latest SMR scandal and a few faulty (as in SMR instead of CMD) hdds in a hundred terabyte ZFS pool and you'll get my frustration level.

PS: The cluster has several dozen terabyte and a lot od nodes. If you have good advice, you're welcome - but please think carefully about this fact.

I might have accidentially vaporized people sending me links with solutions that don't work on large scale TM.

Got offered a new job with a 50% pay increase after 3 interviews, which only one of them was more technical, and it didn't involve any leet code or anything.

My friends coworker tried applying and failed
multiple times previously, so I'm feeling wayyy under qualified because my OOP knowledge sucks, I'm a self taught developer. They asked me more about engineering web solutions - how I would handle a lot of traffic , how I've designed a system where it holds a lot of requests, what do I know about databases, what engines I used and why. I'm very scared to accept, and I like my current company. What do?