Hi there, i'm new to AWS. i've running my python code in Lambda. Basically it's calling an http-request and processing the responding JSON to a CSV file. Now, what is the easiest way to transfer that CSV file to a 3rd party FTP Server?

  • 6
    There is a default ftp lib within python.

    To not let it block look at concurrent.futures lib (i advise the processpoolexecutor, because who cares about slower startup of process in this case)

    Or some queue. Preference is rq (redis based, easy implementation)

    But in general, so many roads to Rome, just pick one
  • 1
    Did you figure it out or need some help?
  • 1
    @rooter hi there, no, i was working on another project, didn't have time to check yet but appreciate your help. I'll let you know as soon as i verified it 👌
  • 1
    @rooter i think a got it.. the only problem is, that i can't test it without a ftp server (not known yet). do you have any preferred Lib for writing tests and/or mock-tests to test the upload function against a mock or something like that?

    and what is the point of using e.g. a queue? i could simply open my local file and stored it with storbinary to a specific path which a navigated to beforehand.
  • 0
    @boese hmm.. Well, if you use a third party ftplib or the included one testing should be their business. If i had to test something like that and really wanted to be sure it 'works' I would create a simple socket listener and if it receives the start of ftp protocol (USER: [username]) it would be enough for me.

    But if that's not enough for you:
    check the asyncssh lib. It's beautiful. So much to plwy with. For building server and client applications.

    Are you talking about sftp or ftp btw?

    About the queue stuff. Well, your application completely blocks during upload probably. That's OK if you have four users and four workers. But with a queue system you could do way more users with less workers. It's not live, but should it be live? The answer is almost always no. I have no idea about your level and project, if you can avoid the queue do it. Implement when performance issues occur. Yagni, kiss, etc

    What solution did you pick?

    Did you also consider mounting a ftp as drive?
  • 1
    @rooter Hey rooter! the last days i worked on this project and finished it last night *wohoo*. it's a simple python app, so i decided to not go with queueing. i don't expect it to block because it fetches 1 api endpoint, processes 1 json file and writes & transfers 1 csv file per day or per week (not yet decieded). but i will def. have a look into it for further projects and for knowledge ofc. i used the built-in ftplib and could establish a connection to a public test-ftp host. so that worked out fine for me as testing. what really grind my gears right now is to decide whtether to host it on AWS Lambda or even on Google Colaboration using AWS Cloud Watch or Google Cloud Scheduler for automatisation (i want to execute it once a day/week automated). do you have any experience in terms of ease of use and costs with any of these services?
  • 1
    @rooter AWS pricing is worth a rant, actually ^^
  • 0
    @boese glad to hear. Sounds like a perfect app (do one thing and do it good). Sadly, I have no experience in that :(.

    I worked for a VPS hosting service and I feel very comfortable to do everything myself. I could host hundreds of apps for the same price. Sadly, maintenance of one can kill the others in theory. So, in terms of deploy and don't look back your approach seems more stable.

    I have this great book laying around and it does cover the aws deployment especially for your kind of app on AWS: https://amazon.com/Python-Microserv...
    I have no idea how up to date it is now. Awesome book in general, covers basic of a big part of development process.

    It's probably better to start a new rant about the pricing since no one is following this old rand 😂 I just don't have knowledge about hosting @ Google / AWS.

    What about digital ocean droplets btw?
  • 1
    @rooter thx for your help, i appreciated it very much!

    also, thx for book and hosting recommendation. didn't know about digital ocean droplets. but the design looks like AWS, doesn't it ?! :D
  • 1
    @boese no idea. Didn't use them. But since they're kinda vpses which gives a lot of freedom I think it's interesting.

    Edit: no problem! I love to help thinking. Sadly, at my previous work I wasn't allowed. I was sad, other people sad, environment sad. So, thank you too 😁
  • 1
    @rooter your're very welcome, glad i could help with asking questions :) enjoy your weekend!
  • 1
    @boese you too! I don't have weekend, I don't work 😁 Those rules don't apply to me 😁
  • 1
    @rooter either way, enjoy saturday and sunday then :D
Add Comment