Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "transcoding"
-
I was messaged on LinkedIn by a recruiter while I was in the UK for my honeymoon. When we got back home to Colorado I called him back and everything went well enough that a tech screen call was set up between four or five guys on the team, and me.
I was expecting to be grilled about various Linux, networking, video transcoding, database, and transaction handling questions and problems, as that was the bulk of the job's description. But instead they just gushed that they'd used software I'd written at previous jobs and loved it.
It was very friendly and they never challenged me (not being arrogant here-- they literally never tested me) and we wound up just talking about, "the job," and about how the work sucked without the tools and apps I'd written.
I got an offer for $30k more than what I asked, the next day.5 -
Okay, story time.
Back during 2016, I decided to do a little experiment to test the viability of multithreading in a JavaScript server stack, and I'm not talking about the Node.js way of queuing I/O on background threads, or about WebWorkers that box and convert your arguments to JSON and back during a simple call across two JS contexts.
I'm talking about JavaScript code running concurrently on all cores. I'm talking about replacing the god-awful single-threaded event loop of ECMAScript – the biggest bottleneck in software history – with an honest-to-god, lock-free thread-pool scheduler that executes JS code in parallel, on all cores.
I'm talking about concurrent access to shared mutable state – a big, rightfully-hated mess when done badly – in JavaScript.
This rant is about the many mistakes I made at the time, specifically the biggest – but not the first – of which: publishing some preliminary results very early on.
Every time I showed my work to a JavaScript developer, I'd get negative feedback. Like, unjustified hatred and immediate denial, or outright rejection of the entire concept. Some were even adamantly trying to discourage me from this project.
So I posted a sarcastic question to the Software Engineering Stack Exchange, which was originally worded differently to reflect my frustration, but was later edited by mods to be more serious.
You can see the responses for yourself here: https://goo.gl/poHKpK
Most of the serious answers were along the lines of "multithreading is hard". The top voted response started with this statement: "1) Multithreading is extremely hard, and unfortunately the way you've presented this idea so far implies you're severely underestimating how hard it is."
While I'll admit that my presentation was initially lacking, I later made an entire page to explain the synchronisation mechanism in place, and you can read more about it here, if you're interested:
http://nexusjs.com/architecture/
But what really shocked me was that I had never understood the mindset that all the naysayers adopted until I read that response.
Because the bottom-line of that entire response is an argument: an argument against change.
The average JavaScript developer doesn't want a multithreaded server platform for JavaScript because it means a change of the status quo.
And this is exactly why I started this project. I wanted a highly performant JavaScript platform for servers that's more suitable for real-time applications like transcoding, video streaming, and machine learning.
Nexus does not and will not hold your hand. It will not repeat Node's mistakes and give you nice ways to shoot yourself in the foot later, like `process.on('uncaughtException', ...)` for a catch-all global error handling solution.
No, an uncaught exception will be dealt with like any other self-respecting language: by not ignoring the problem and pretending it doesn't exist. If you write bad code, your program will crash, and you can't rectify a bug in your code by ignoring its presence entirely and using duct tape to scrape something together.
Back on the topic of multithreading, though. Multithreading is known to be hard, that's true. But how do you deal with a difficult solution? You simplify it and break it down, not just disregard it completely; because multithreading has its great advantages, too.
Like, how about we talk performance?
How about distributed algorithms that don't waste 40% of their computing power on agent communication and pointless overhead (like the serialisation/deserialisation of messages across the execution boundary for every single call)?
How about vertical scaling without forking the entire address space (and thus multiplying your application's memory consumption by the number of cores you wish to use)?
How about utilising logical CPUs to the fullest extent, and allowing them to execute JavaScript? Something that isn't even possible with the current model implemented by Node?
Some will say that the performance gains aren't worth the risk. That the possibility of race conditions and deadlocks aren't worth it.
That's the point of cooperative multithreading. It is a way to smartly work around these issues.
If you use promises, they will execute in parallel, to the best of the scheduler's abilities, and if you chain them then they will run consecutively as planned according to their dependency graph.
If your code doesn't access global variables or shared closure variables, or your promises only deal with their provided inputs without side-effects, then no contention will *ever* occur.
If you only read and never modify globals, no contention will ever occur.
Are you seeing the same trend I'm seeing?
Good JavaScript programming practices miraculously coincide with the best practices of thread-safety.
When someone says we shouldn't use multithreading because it's hard, do you know what I like to say to that?
"To multithread, you need a pair."18 -
Really just an average week.
Just feel I need a bit of venting. (:
@meet: (monday)
- mgr: we need video transcoding and VOD ASAP.
- dev: on what server? It's expensive, especially without a GPU.
- mgr: prod is beefy. Put it there.
- dev: everything else is gonna crawl then.
- mgr: you have till the end of this week.
@demo (Friday)
- dev: k, it's ready.
- mgr: Why is everything slow??!
- dev: transcoding. Expensive.
- mgr: Why do we transcode? Never said I wanted transcode!
Can't we upload to YT?
- dev: ...yes. But will then each customer that wants VOD will need to setup YT studio and provide an endpoint and stream key.
- mgr: OK. But we're now behind schedule because of this and the customers will not be pleased.
- dev: oh, didn't know we're into gaming.
- mgr: ???
- dev: nvm, see you Monday.
...
Later Friday evening
...
*ding* mgr has added 5 new tasks to your list.
*ding* mgr subtracted 30 points from you.
reason: deadline over due.
Ya ya, the usual shenanigans.
Time to mute for the weekend.11 -
Built a Media server using emby and then realized that one of my old TV shows with only 5 seasons was taking 129GB of storage space and I would need to transcode them, did my research and decided to use VP9 for the codec (YouTube, native browser support, etc.) found that it reduced my shows size to 51GB with great quality still. And then found out that emby doesn't allow direct play of VP9 on browsers and it's transcoding back to H264 Everytime I watch it. Submitted a issues, we'll see what happens now.3
-
My current "file/media server" is a crappy old falling apart windows box with a stupid mismash of internal and external drives with no redundancy. That sucks for a number of reasons, so planning on dropping around a grand or so (including drives) on doing it properly.
Space requirement would be around 20TB-ish of usable space, with 1 disk's worth of redundancy. That can include a newish 5TB drive I have lying around however. Would also run either Plex / Jellyfin, so some horsepower for transcoding would be nice (but no need for more than a single 1080 stream at once.) 24/7 operation, so don't want anything too power hungry.
Current (loose) thinking on the hardware side is an AM4 board and a reasonably low end CPU, 3x8TB WD golds. Software side, probably CentOS, then mergerfs + snapraid. Anyone got any insight as to other options? Hardware not my speciality in particular, so open to suggestions.14
