Attributeerror: 'nonetype' object has no attribute 'get_text' when webscraping

Question

0.00/5 (No votes)

See more:

The aim of this code is o scrape data from Search Jobs | CareerJunction[^] and allow the user to input the job title they are looking for and for it to display the job title with its attributes like salary, date posted,etc.

import requests
from bs4 import BeautifulSoup

def job_details(job_title):
    base_url="https://www.careerjunction.co.za/"
    search_url="https://www.careerjunction.co.za/jobs/results?keywords=&autosuggestEndpoint=%2Fautosuggest&location=0&category=&btnSubmit="
    job_search=f"{search_url}, keywords={job_title.replace(' ','%20')}"
    
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"}
    search_url_response=requests.get(search_url, headers=headers)
    if search_url_response.status_code != 200:
        print(f"Unable to fetch page. Status code{search_url_response}")
        return
    
    search_content=search_url_response.text
    soup=BeautifulSoup(search_content,"html.parser")

    jobs_c1=soup.find_all("div", class_="job-result-title")
    for jobs_c1 in jobs_c1:
        title=jobs_c1.find("h2", class_="href").get_text(strip=True) 
        recruiter_name=jobs_c1.find("h3", class_="href").get_text(strip=True) 

    jobs_c2=soup.find_all("div", class_="job-result-overview cjun-serp")
    for jobs_c2 in jobs_c2:
        salary=jobs_c2.find("ul", class_="salary").get_text(strip=True) 
        position=jobs_c2.find("ul", class_="position").get_text(strip=True) 
        location=jobs_c2.find("ul", class_="location").get_text(strip=True)
        date_posted=jobs_c2.find("ul", class_="updated-time").get_text(strip=True)

    print(f"Job Title: {title}")
    print(f"Recruiter name: {recruiter_name}")
    print(f"Salary: {salary}")
    print(f"Position: {position}")
    print(f"Location: {location}")
    print(f"Date Posted: {date_posted}")


job_title=input(" Enter the job you are looking for: ")
job_details(job_title)

What I have tried:

I've tried alterting the get_text to gettext and getText but still the code isnt running and i get an attribute error
The aim of this code is o scrape data from Search Jobs | CareerJunction[^] and allow the user to input the job title they are looking for and for it to display the job title with its attributes like salary, date posted,etc.

Posted 24-Jun-24 4:43am

Lumko Mtengwane

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Answer 1 · 2024-06-24T06:49:00

You have not explained where the error occurs, but I assume it is on this line:

Python

title=jobs_c1.find("h2", class_="href").get_text(strip=True)

The error message tells you that the call to find did not return any data, so there is no text to be got. You should always check the response/return codes for method calls, before trying to use what you think may be returned. Do not just assume that such calls always succeed.

Pete O'Hanlon · Answer 2 · 2024-06-24T06:47:00

The problem isn't that you're using get_text (or whatever function). The problem is that the code has is returning a NoneType (Python's version of null). In other words, you have failed to find a particular element on the web page, and you are attempting to get the text from something that doesn't exist. You should check to see if the value is None before attempting to access the element

Python

titleItem=jobs_c1.find("h2", class_="href")
if title != None
    title = titleItem.get_text(strip = True)

Repeat this through your code as necessary.

Attributeerror: 'nonetype' object has no attribute 'get_text' when webscraping

2 solutions

Solution 2

Solution 1

Add your solution here

Preview 0