Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
Search - "scrapy"
-
Fixing bugs in programming is like scoring a touchdown when you really didn't even know you had the ball, or where the end zone is.1
-
I'm hoping it's not just me, but whenever I'm using a website or mobile app, and see something badly designed I get very angry...2
-
I get an email about an hour before I get into work: Our website is 502'ing and our company email addresses are all spammed! I login to the server, test if static files (served separately from site) works (they do). This means that my upstream proxy'd PHP-FPM process was fucked. I killed the daemon, checked the web root for sanity, and ran it again. Then, I set up rate limiting. Who knew such a site would get hit?
Some fucking script kiddie set up a proxy, ran Scrapy behind it, and crawled our site for DDoS-able URLs - even out of forms. I say script kiddie because no real hacker would hit this site (it's minor tourism in New Jersey), and the crawler was too advanced for joe shmoe to write. You're no match for well-tuned rate-limiting, asshole!1 -
It's cool finally seeing real world feedback for something you developed and programmed yourself, I'm still in college and I'm trying to start a company with some software I came up with, and it's being taken very well from potential clients!🙏🏻3
-
Lab needs a crawler to download some assets, none of my business though
But why not
Haven't touched crawler for two years
Google for latest state of art
Found scrapy
I have to define a class for a crawling script?
Got scared
Went back to beautifulsoup and request
Got the job done in 20 mins
Fuck yeah6 -
I spent 4 months in a programming mentorship offered by my workplace to get back to programming after 4 years I graduated with a CS degree.
Back in 2014, what I studied in my first programming class was not easy to digest. I would just try enough to pass the courses because I was more interested in the theory. It followed until I graduated because I never actually wrote code for myself for example I wrote a lot of code for my vision class but never took a personal initiative. I did however have a very strong grip on advanced computer science concepts in areas such as computer architecture, systems programming and computer vision. I have an excellent understanding of machine learning and deep learning. I also spent time working with embedded systems and volunteering at a makerspace, teaching Arduino and RPi stuff. I used to teach people older than me.
My first job as a programmer sucked big time. It was a bootstrapped startup whose founder was making big claims to secure funding. I had no direction, mentorship and leadership to validate my programming practices. I burnt out in just 2 months. It was horrible. I experienced the worst physical and emotional pain to date. Additionally, I was gaslighted and told that it is me who is bad at my job not the people working with me. I thought I was a big failure and that I wasn't cut out for software engineering.
I spent the next 6 months recovering from the burn out. I had a condition where the stress and anxiety would cause my neck to deform and some vertebrae were damaged. Nobody could figure out why this was happening. I did find a neurophyscian who helped me out of the mental hell hole I was in and I started making recovery. I had to take a mild anti anxiety for the next 3 years until I went to my current doctor.
I worked as an implementation engineer at a local startup run by a very old engineer. He taught me how to work and carry myself professionally while I learnt very little technically. A year into my job, seeing no growth technically, I decided to make a switch to my favourite local software consultancy. I got the job 4 months prior to my father's death. I joined the company as an implementation analyst and needed some technical experience. It was right up my alley. My parents who saw me at my lowest, struggling with genetic depression and anxiety for the last 6 years, were finally relieved. It was hard for them as I am the only son.
After my father passed away, I was told by his colleagues that he was very happy with me and my sisters. He died a day before I became permanent and landed a huge client. The only regret I have is not driving fast enough to the hospital the night he passed away. Last year, I started seeing a new doctor in hopes of getting rid of the one medicine that I was taking. To my surprise, he saw major problems and prescribed me new medication.
I finally got a diagnosis for my condition after 8 years of struggle. The new doctor told me a few months back that I have Recurrent Depressive Disorder. The most likely cause is my genetics from my father's side as my father recovered from Schizophrenia when I was little. And, now it's been 5 months on the new medication. I can finally relax knowing my condition and work on it with professional help.
After working at my current role for 1 and a half years, my teamlead and HR offered me a 2 month mentorship opportunity to learn programming from scratch in Python and Scrapy from a personal mentor specially assigned to me. I am still in my management focused role but will be spending 4 hours daily of for the mentorship. I feel extremely lucky and grateful for the opportunity. It felt unworldly when I pushed my code to a PR for the very first time and got feedback on it. It is incomparable to anything.
So we had Eid holidays a few months back and because I am not that social, I began going through cs61a from Berkeley and logged into HackerRank after 5 years. The medicines help but I constantly feel this feeling that I am not enough or that I am an imposter even though I was and am always considered a brilliant and intellectual mind by my professors and people around me. I just can't shake the feeling.
Anyway, so now, I have successfully completed 2 months worth of backend training in Django with another awesome mentor at work. I am in absolute love with Django and Python. And, I constantly feel like discussing and sharing about my progress with people. So, if you are still reading, thank you for staying with me.
TLDR: Smart enough for high level computer science concepts in college, did well in theory but never really wrote code without help. Struggled with clinical depression for the past 8 years. Father passed away one day before being permanent at my dream software consultancy and being assigned one of the biggest consultancy. Getting back to programming after 4 years with the help of change in medicine, a formal diagnosis and a technical mentorship.3 -
tldr; selenium-java (my newest learned tool) vs beautifulsoup4 (my most experience with) or scrapy(average experience, mediocre ability with). Which should I use if allowed to use any for web scrapeing assignment
We were explicitly told we can use anything we know from class or self study (slight bonus for self study implementations) for the group project, but would it be OK/fair for me to use beautifulsoup4 or scrapy to pull the data from the assigned site rather than the selenium-java we were taught in class
If I did use bs4 or scrapy my group wouldn't be able to edit if needed but the data collection is only a small (if immensely important) part of the assignment and I'd have the bs4 script done a lot quicker than with selenium which I have learned more recently (for class) and have less experience with9 -
It goes back two years ago, i was writing web crawlers with scrapy. i don't remember how long i worked, but i think it took a full day
why: because, web crawling is so much fun and also i was young and stupid -
Does anyone else's notifs glitch out on here? I keep clearing them but every time I load it back up my notifications are back - currently up to 16 and they won't go away10
-
Do you prefer audiobooks? Are you an active medium reader? Do you want audio for the medium articles you read? Are you out of your free medium articles?😢 My Scrapy is here for the rescue.💸
This is a simple application of web scraping, it scrapes the articles of medium and allows you to read or hear the article. If you use this on computer there will be a number of accents in the option.
The audio feature is provided only to the premium medium users, so here comes My Scrapy to save your 5$/month. 💸
.
Tech Stack used :
Python, beautiful soup, Django, speech synthesis
PS: This application was built for educational purpose.
Fun Fact: You can still read any medium articles if they are asking you to upgrade, you must be wondering how? Well, copy the link of the article and browse it in incognito mode on any browser or sign out and read it.😂🤣
githublink:
https://github.com/globefire/...
demo link:
https://youtube.com/watch/...
instagram link:
https://instagram.com/p/...3 -
Are you out of your free medium articles?😢 My Scrapy is here for the rescue.💸
This is simple application of web scraping, it scrapes the articles of medium and allows you to read or hear the article. If you use this on computer there will be a number of accents in the option.
The audio feature is provided only to the premium medium users, so here comes My Scrapy to save your 5$/month. 💸
.
Tech Stack used :
Python, beautiful soup, Django, speech synthesis
.
PS: This application was built for educational purpose and the source code for this application is not open sourced anywhere.
Fun Fact : You can still read any medium articles if they ask you to upgrade, you must be wondering how? Well, copy the link of the article and browse it in incognito mode on any browser.😂🤣
Try the app and lemme know if you liked it:
https://mymediumscraper.herokuapp.com/...4