Python Selenium Skips Over Necessary Elements
PLEASE DONT DOWNVOTE, THIS QUESTION IS DIFFERENT FROM PREVIOUS ONE, IM USING DIFFERENT LOGIC HERE Im trying to iterate over all user reviews('partial_entry' class) from this page h
Solution 1:
One tip for removing chromedriver path in every script.Put chromedriver.exe in C:\Python27\Scripts than you don't have to put the chromedriver path in every script than just use driver = webdriver.Chrome()
I am running this code:
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.maximize_window()
url="https://www.tripadvisor.com/Airline_Review-d8729164-Reviews-Cheap-Flights-or560-TAP-Portugal#REVIEWS"
driver.get(url)
ctr=0for j in driver.find_elements_by_xpath("//div[@class='wrap']"): # FIND ALL REVIEW ELEMENTSfor ent in j.find_elements_by_xpath('.//p[@class="partial_entry"]'): # FIND REVIEW TEXT# FIRST CHECK IF TRANSLATION IS AVAILABLE (I.E. NON ENGLISH COMMENTS)if j.find_elements_by_css_selector('#REVIEWS .googleTranslation>.link'):
#print 'NOW PRINTING TRANSLATED COMMENTS'
gt= driver.find_elements(By.CSS_SELECTOR,"#REVIEWS .googleTranslation>.link")
size=len(gt)
while (ctr<size):
for i in gt:
try:
ifnot i.is_displayed():
continue
driver.execute_script("arguments[0].click()",i)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, ".//span[@class = 'ui_overlay ui_modal ']//div[@class='entry']")))
com= driver.find_element_by_xpath(".//span[@class = 'ui_overlay ui_modal ']//div[@class='entry']")
print com.text
print"++" * 60
time.sleep(5)
driver.find_element_by_class_name("ui_close_x").click()
time.sleep(5)
#loop+=1except Exception as e:
print"skipped"pass
ctr+=1# COMMENT ALREADY IN ENGLISH, PRINT AS IT ISelse:
print ent.text
print"="*60
driver.quit()
the output I am getting is:
Quite comfortable for the economy class, with a friendly staff and good service. The food is good but could still be better.
============================================================
The pilot was amazing, soft takeoff, soft landing (even with ruf weather), very nice staff with amazing portuguese food and wine. The only downsize was the interior condition, although clean and without scratches or so you could see that is already aged. Appart from that all was good.
============================================================
Speedy check in process was very accurate and precise. They allowed cabin to be booked into the hold with no additional charges. Boarding was efficient and timely. the seats were very comfortable. Wide enough to fit me fairly comfortably with armrests that were able to lift during the flight. The really stand out thing forme was the leg space....
============================================================
My country's flag airline, It has struggle to survive in a hard economic cycle. Clever choice of unique African and south american cities, guarantied its continuity.~ Do not expect a exquisite food, alcoholic drinks, down to beer and wine, forget white spirits. Good safety record. Pilots well trained, good maintenance. I have flight TAP for the last 40 odd years...
============================================================
Our first trip to Europe on a long flight both ways. The flight TO Rome was good. I am tall and have back issues, and thank God we were able togetexit row seats. This made all the difference in the world. The food served was fair to good. There were movies offered which helped pass the time and...
============================================================
On my change my flight without asking my opinion or offer another solution without paying extra I stay more than 10 hours in boarding of room I have the urge to have something to eat I haven not even able to rest after my flight c is inadmissible night I no longer would resume this company and would not advise a person totake
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
A little apprehensive before but quickly lifted. Very welcome and good service from the PNC, hot meal and good even for this short flight (1h50). Good punctuality and boarding more efficient
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Everything normal. Aircraft clean and almost full. Embarking on time, regular. Arrive slightly earlier. friendly and courteous staff. On board it was given a snack.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
In the recent past I have traveled a few times from Venice to Lisbon andfrom Venice to Oporto via Lisbon. Good facilities on land and aboard; friendly service, clean air, punctuality and competitive rates. recommended
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Sympathy and competence. The company strives to make passengers as comfortable as possible.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Update for handling the things in comment:
- First install ntkl using command "pip install nltk"
- Once installation complete
- Open python shell ..i.e, Idle
- Type command: import nltk
- now type: nltk.download()
- The UI will open: click on Models...search for punkt & click on download
- After this ...click on Corpora....search for stopwords & click on download
One these installation are done: run following program:
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from nltk import word_tokenize
from nltk.corpus import stopwords
defdetect_lang(text):
lang_ratios = {}
tokens = word_tokenize(text)
words = [word.lower() for word in tokens]
for language in stopwords.fileids():
stopwords_set = set(stopwords.words(language))
words_set = set(words)
common_elements = words_set.intersection(stopwords_set)
lang_ratios[language] = len(common_elements)
returnmax(lang_ratios, key=lang_ratios.get)
driver = webdriver.Chrome()
driver.maximize_window()
url="https://www.tripadvisor.com/Airline_Review-d8729164-Reviews-Cheap-Flights-or570-TAP-Portugal#REVIEWS"
driver.get(url)
ctr=0
time.sleep(5)
defexpand_reviews(driver):
# TRYING TO EXPAND REVIEWS (& CLOSE A POPUP)try:
driver.find_element_by_class_name("moreLink").click()
except:
print"err"try:
driver.find_element_by_class_name("ui_close_x").click()
except:
print"err2"try:
driver.find_element_by_class_name("moreLink").click()
except:
print"err3"# # FIRST EXPAND THE REVIEWS BY CLICKING "MORE" BUTTON
expand_reviews(driver)
time.sleep(10)
for ent in driver.find_elements_by_xpath('.//div[@class="entry"]/p[1]'): # FIND REVIEW TEXT
lang = detect_lang(ent.text)
if (lang == 'english'):
print ent.text
print"=="*30else:
if driver.find_elements_by_css_selector('#REVIEWS .googleTranslation>.link'):
gt= driver.find_elements(By.CSS_SELECTOR,"#REVIEWS .googleTranslation>.link")
size=len(gt)
while (ctr<(size/2)):
for i in gt:
try:
ifnot i.is_displayed():
continue
driver.execute_script("arguments[0].click()",i)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, ".//span[@class = 'ui_overlay ui_modal ']//div[@class='entry']")))
com= driver.find_element_by_xpath(".//span[@class = 'ui_overlay ui_modal ']//div[@class='entry']")
print com.text
print"++" * 60
time.sleep(5)
driver.find_element_by_class_name("ui_close_x").click()
time.sleep(5)
#loop+=1except Exception as e:
print"skipped"pass
ctr+=1This will print following output:
Speedy check in process was very accurate and precise. They allowed cabin to be booked into the... read more
============================================================
Very pleasant flight, excellent service on board andon the ground, the best seats in the Buisness Classand Top Food and drinks during the flight.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Perfect atendimento.Bom care of Commissioners and Commissioners, punctuality. Good movies offered.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Overall, a good flight! Time (departure and arrival). Enough time for the change to Lisbon. Very nice crew!
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
It was a flight noturno.Teve strong turbulence and I could not dormir.Rezei all night. After all it was a decent trip. only regret the discomfort of the aircraft but praise the good atendimento.Toda the crew was very kind and helpful.The journey back was quieter.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The food really isnot the best to tell the truth, I could not even eat. But the service is very good.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Kindness, professionalism, and willingness on the part of the crew: good landing and includes drinks and light dinner
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
On9 October flown with this company. By delayed entering the unit departed late. Atmosphere Loos routine operation. The evening meal consisted of a tuna sandwich and a liquid plum in plastic vial. A choice of meat or cheese was not there. Formeand many others so no meal on this flight. Downright depressing.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Speedy check in process was very accurate and precise. They allowed cabin to be booked into the hold with no additional charges. Boarding was efficient and timely. the seats were very comfortable. Wide enough to fit me fairly comfortably with armrests that were able to lift during the flight. The really stand out thing forme was the leg space. I'm 6ft 4in and I was able to fully extend my legs under the chair in front of me. if for this reason alone I have started looking for other destinations they fly to. In flight entertainment was none existent but then it was only a short haul flight so I won't fault them. Food snack and beverages were included in the price. The in flight attendants were professional, courteous and well presented. I will definitely use them again in the future!
============================================================
My country's flag airline, It has struggle to survive in a hard economic cycle. Clever choice of unique African and south american cities, guarantied its continuity.~Donot expect a exquisite food, alcoholic drinks, down to beer and wine, forget white spirits. Good safety record. Pilots well trained, good maintenance. I have flight TAP for the last 40 odd years I have seen many faces and crisis, however customer service really Bad.
============================================================
Our first trip to Europe on a long flight both ways. The flight TO Rome was good. I am tall and have back issues, and thank God we were able togetexit row seats. This made all the difference in the world. The food served was fair to good. There were movies offered which helped pass the time and blankets because the cabin got very cold. We had to sit in Lisbon Airport for six hours to complete our journey. It was tiring but still the trip was good (fun even)
The flight back home was notas pleasant. The first leg from Rome to Lisbon was horrible. The woman in front ofme kept slamming her seat up against my legs and telling me she had to lay back andto put my tray table up. The flight attendant SAW this happen and did nothing. I found the crew to be very unfriendly on all legs of the flight. They were not warm and friendly and even bordered on rude at some point.
I would probably fly this airline again because the price was right and it was acceptable....
============================================================
Post a Comment for "Python Selenium Skips Over Necessary Elements"