Do all the things like ++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatarSign Up
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple APILearn More
rooter102310dThere is a default ftp lib within python.
To not let it block look at concurrent.futures lib (i advise the processpoolexecutor, because who cares about slower startup of process in this case)
Or some queue. Preference is rq (redis based, easy implementation)
But in general, so many roads to Rome, just pick one
rooter10238dDid you figure it out or need some help?
boese927d@rooter i think a got it.. the only problem is, that i can't test it without a ftp server (not known yet). do you have any preferred Lib for writing tests and/or mock-tests to test the upload function against a mock or something like that?
and what is the point of using e.g. a queue? i could simply open my local file and stored it with storbinary to a specific path which a navigated to beforehand.
rooter10237d@boese hmm.. Well, if you use a third party ftplib or the included one testing should be their business. If i had to test something like that and really wanted to be sure it 'works' I would create a simple socket listener and if it receives the start of ftp protocol (USER: [username]) it would be enough for me.
But if that's not enough for you:
check the asyncssh lib. It's beautiful. So much to plwy with. For building server and client applications.
Are you talking about sftp or ftp btw?
About the queue stuff. Well, your application completely blocks during upload probably. That's OK if you have four users and four workers. But with a queue system you could do way more users with less workers. It's not live, but should it be live? The answer is almost always no. I have no idea about your level and project, if you can avoid the queue do it. Implement when performance issues occur. Yagni, kiss, etc
What solution did you pick?
Did you also consider mounting a ftp as drive?
@rooter Hey rooter! the last days i worked on this project and finished it last night *wohoo*. it's a simple python app, so i decided to not go with queueing. i don't expect it to block because it fetches 1 api endpoint, processes 1 json file and writes & transfers 1 csv file per day or per week (not yet decieded). but i will def. have a look into it for further projects and for knowledge ofc. i used the built-in ftplib and could establish a connection to a public test-ftp host. so that worked out fine for me as testing. what really grind my gears right now is to decide whtether to host it on AWS Lambda or even on Google Colaboration using AWS Cloud Watch or Google Cloud Scheduler for automatisation (i want to execute it once a day/week automated). do you have any experience in terms of ease of use and costs with any of these services?
@boese glad to hear. Sounds like a perfect app (do one thing and do it good). Sadly, I have no experience in that :(.
I worked for a VPS hosting service and I feel very comfortable to do everything myself. I could host hundreds of apps for the same price. Sadly, maintenance of one can kill the others in theory. So, in terms of deploy and don't look back your approach seems more stable.
I have this great book laying around and it does cover the aws deployment especially for your kind of app on AWS: https://amazon.com/Python-Microserv...
I have no idea how up to date it is now. Awesome book in general, covers basic of a big part of development process.
It's probably better to start a new rant about the pricing since no one is following this old rand 😂 I just don't have knowledge about hosting @ Google / AWS.
What about digital ocean droplets btw?