so apparently devrant has an image dimension limitation~
compressing it from a 9mb one to 2.7mb still gives an error on jpg with the resolution 4156x2258!
@dfox is it just me or a random bug?

I wish there was a way to not need to compress the image. Not saying @dfox's server should be serving 50mb images but the compression often looks pretty bad.

Seeing as DevRant runs on PHP, I imagine devrant is using either GD or ImageMagick. If you're using GD, try ImageMagick it's a lot more robust. There's quite a few compression scheme that ImageMagick provides: "None, BZip, Fax, Group4, JPEG, JPEG2000, Lossless, LZW, RLE or Zip." I definitly suggest testing them out to see which gives the best looking (least amount of degradation) results for several different image types, and dimensions.

Out of curiosity, I ran a whois on DevRant.net which tells me their registrar is transip.nl, and a "WhosHostingThis" query tells me it's also their hosting provider. Which is weird because they're based in the Europe but @dfox and his partner apparently are in New York (link to Hexical Labs in the footer of the webapp brings me

to a page with both of your linkedin pages; don't worry I'm not stalking you guys). So I'm not sure why there using that european host although their VPS prices look pretty nice.

If you guys ever want to change hosts, I would suggest AWS. It's more expensive. You can write a script that spins up/down more instances based on your current load. Thats a lot quicker than provisioning a VPS at a smaller host; an hour or two vs a minute or two.

They provide dozens of other services tied into the EC2 VPS service. They provide managed RDBMS so you don't have to worry about tweaking or hitting hardware limits. Using something like S3 + CloudFront (file hosting + global CDN) to serve these image files would be quite a bit faster, and free up your VPS from needing to serve images, which is probably not an insignificant load.

They offer auto-compression for images up to 10MB, which apparently about how large of files DevRant allows you to upload now.

@rozzzly wrong URL :)

I'm looking into this. We've seen this issue with a few images.

We use GD right now. I had switched to ImageMagick at one point, for avatars, and it performed significantly slower than GD so we had to stick with GD.

At this point we can also up the file size limit, but in many cases it seems that wasn't actually what made the upload fail. This was just an arbitrary limit and our web server can really handle much higher.

@dfox try checking the php.ini for max_post and max upload that usually fixed it should be in /etc/php/php-fpm if you use nignx for web side if not it is is in the other folder in the PHP folder and don't forget to up the upload size in the web hosting software to

So for avatars, I imagine you're using GD to composite a combination of PNGs on top of eachother. Like one of x skin colors, then on top of that you place one of the y hair colors? That's something that GD can do just fine, all it's doing is laying pixels on top of eachother.

I'd imagine ImageMagick is slower because because PHP has to exec a command to run ImageMagick via the shell, and blocks until that finishes.

IIRC your stack doesn't have any language capable of a concurrent job queue. If I were you, I would set up a NodeJs app that runs as daemon. You could then use something like redis to hold a list files that need to compressed/resized/etc. PHP writes original image to the disk, then uses redis to append a task to the queue. The Node daemon is listening to that queue, when something new comes in, it pops it off the "waiting" queue, takes the uri of the file and then preforms the desired operations, the pops the job out of that queue, puts it into a list of finished ones.

If you're using NodeJS you have a lot of options for image processing libs.

This looks really nice:
https://github.com/lovell/sharp
It has bindings to native multi-threaded c lib so you'll get real good preformance.

And theres also a project that forked ImageMagick in 2002 and claims to be faster&lighter, and is used by flickr
http://www.graphicsmagick.org/

Only issue I can see depends on the flow of your upload handling script. You couldn't be certain that by the next time the page refreshes that the image would be done. With GD you can be certain because it blocks the response until finished. You would have to do something like: when rendering subsequent responses that want that image check either the db row or the filesystem to see if the file is there. If not put a placeholder saying: processing... or something.

If you want the user to get the satisfaction of the processed image just appearing when done, you then poll an api endpoint via XHR that checks job's status via redis

@rozzzly Backseat programming :p .

they already use aws cloudfront for serving images, check a record of img.devrant.io which will lead to a cloudfront subdomain.

@skonteam guilty.

@rjcrystal hmm thats even odder considering they dont use EC2

@rjcrystal is correct. All of our *.devrant.io image urls (avatars, img) are aliases for CloudFront which also uses S3 in one way or another to fetch images. In the case of avatars, if the image doesn't exist on CloudFront or S3, it's generated by PHP on the fly and then cached in both CloudFront and S3 (S3 so other CloudFront edges don't need a PHP regen).

@rozzzly thanks for the suggestions! Your guess of how avatars are generated is pretty much spot on :) Though I can't think of any way to do that asynchronously because like you already pointed out, most images are needed right away for the requesting user and generally should be blocking.

As for queuing, we actually do a ton of queuing right now (async jobs) with SQS to the tune of a few hundred thousand jobs per day.

I don't think anything in particular for image uploads really needs to be async. I think as a user, if I'm uploading a 15mb image, it wouldn't be unexpected for there to be a bit more processing time involved. I'll do some tests to see how long it actually is, but the original limit wasn't imposed because of slowness, more because I didn't think people using mobile devices would want to upload images that big, but I see now I was wrong so we'll adjust :)

@rozzzly you don't need ec2 to use cloudfront. You can just put your assets in s3 or any other external origin and you're good to go.