I am getting error in scrapping content through selenium python
I am scrapping the title of jobs results on https://www.indeed.ae/jobs-in-dubai through selenium. i think .text is not working. i am running the code through selenium which go to main website, enter selective keyword and then scrap all titles from result. but i am getting error, how can i solve this error
here is my code
import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.common.keys import Keys Path = "C:\Program Files (x86)\chromedriver.exe" driver = webdriver.Chrome(Path) driver.get("https://indeed.ae/") print(driver.title) search = driver.find_element_by_name("l") search.send_keys("Dubai") search.send_keys(Keys.RETURN) try: td = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "resultsCol")) ) divs = td.find_elements_by_tag_name("div") for div in divs: header = div.find_element_by_class_name("title") print(header) finally: driver.quit() driver.quit() and i am getting following error
Job Search | Indeed Traceback (most recent call last): File "C:/Users/hp/Desktop/python projects/selenium-pycharm/selenium-bot.py", line 24, in <module> header = div.find_element_by_class_name("title") File "C:\Users\hp\Desktop\python projects\selenium-pycharm\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 398, in find_element_by_class_name return self.find_element(by=By.CLASS_NAME, value=name) File "C:\Users\hp\Desktop\python projects\selenium-pycharm\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 659, in find_element {"using": by, "value": value})['value'] File "C:\Users\hp\Desktop\python projects\selenium-pycharm\venv\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute return self._parent.execute(command, params) File "C:\Users\hp\Desktop\python projects\selenium-pycharm\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Users\hp\Desktop\python projects\selenium-pycharm\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".title"} (Session info: chrome=83.0.4103.116) Process finished with exit code 1 thanks in advance
You can not find title because you get all of the div from the resultsCol. which means some div have title and some not.
Try this :
try: td = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "resultsCol")) ) divs = td.find_elements_by_tag_name("div") #print(divs) for div in divs: try: header = div.find_element_by_class_name("title") print(header.text) except: continue finally: driver.quit() driver.quit() which give title as output :
Receptionist Administrative Assistant/ Document Controller RECEPTIONIST ADMIN OFFICER IN UAE Data Entry Assistant (Fresh Graduate) Receptionist Replenishment Associate - Light Household - Hypermarket DOCUMENT CONTROLLER School Administrative Assistant - Dubai ACCOUNTANT 
