Selenium and BeautifulSoup 4 combination broke .find_all, making variable undefined
10:42 14 Jun 2026

I'm combining Selenium with code i had made with BeautifulSoup4 to scrape a page for prices, and even though the code worked fine with only BeautifulSoup4, .find_all stopped working and leaves one of my variables undefined when i implemented Selenium. My goal is for all prices to print and be added to the list, where it will then print all prices combined.

from bs4 import BeautifulSoup
import sched, time
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By

pets = [] 

driver= Chrome()
 
driver.get("https://starpets.gg/adopt-me/shop/pet/hot_doggo/17996")
driver.implicitly_wait(5)
cookiedecl = driver.find_element(By.CLASS_NAME, "cookie__button._button_1fum3_1._button__secondary_1fum3_413._button_mm_1fum3_134._button_fullWidth_1fum3_268._button__text_upper_1fum3_73._button__text_ellipsis_1fum3_76.cookie__button").click()
time.sleep(5)
PetAges = driver.find_element(By.CLASS_NAME, "_container_1dlh7_1._size__xl_1dlh7_72")
PetMenu = PetAges.find_element(By.XPATH, '//*[@title="Select age"]').click()
time.sleep(5)
NewbFind = driver.find_element(By.CLASS_NAME, "_container_wdj39_1._size__s_wdj39_87")
NewbClick = NewbFind.find_element(By.XPATH, '//*[text()="Newborn"]').click()
Petdoc = BeautifulSoup(driver.page_source, "html.parser")
time.sleep(10)
Pettag1 = Petdoc.select("p._text_j98bt_1._text__size_2xl_j98bt_55._text__nowrap_j98bt_7._text__weight_bold_j98bt_83._text__style_normal_j98bt_95._text__decoration_normal_j98bt_104._info__price__count_ilro6_87, span.itemprop.price") 
for PetbestPrice in Pettag1: 
 Petprice1 = float(PetbestPrice.text.replace("$", "")) 
 pets.append(Petprice1) 
print(Petprice1)

Pettag2 = Petdoc.find_all("div", class_ = "_text_j98bt_1 _text__size_m_j98bt_40 _text__weight_bold_j98bt_83 _text__style_normal_j98bt_95 _text__decoration_normal_j98bt_104 _content-price_1xjd5_90")
for PetotherPrice in Pettag2:
 Petprice2 = float(PetotherPrice.text.split(' ')[0])
 pets.append(Petprice2)
print(Petprice2)
print(pets)
driver.close()

executing this gives me the error NameError: name 'Petprice2' is not defined. Did you mean: 'Petprice1'? when it tries to execute print(Petprice2)

I tried using the time.sleep(10) once i handover the page source to BeautifulSoup in case it was a loading issue, however, it still only prints the first price.

I also tried replacing the Petprice2 = float(PetotherPrice.text.split(' ')[0]) with Petprice2 = float(PetotherPrice.text.replace("$", "")) but it had the same result.

Why did this become an issue once selenium was incorporated? Is there a way to fix it (and possibly make my code better in the process?)

python selenium-webdriver beautifulsoup