Can't Scrape All Of Ul Tags From A Table

I'm trying to scrape all of proxy ips from this site : https://proxy-list.org/english/index.php but i can only get one ip at most here is my code : from helium import * from bs4

Solution 1:

from bs4 import BeautifulSoup
import requests

url = 'https://proxy-list.org/english/index.php'

pagecontent = requests.get(url)
soup = BeautifulSoup(browser.pagecontent, 'html.parser')
maintable = soup.find_all('div', {'class':'table'})
for div_element  in maintable:
    rows = div_element.find_all('li', class_='proxy')
    for ip in rows:
        print(ip.text)

Solution 2:

If I get your question right, the following is one of the ways how you can fetch those proxies using requests module and Beautifulsoup library:

import re
import base64
import requests
from bs4 import BeautifulSoup

url = 'https://proxy-list.org/english/index.php'defdecode_proxy(target_str):
    converted_proxy = base64.b64decode(target_str)
    return converted_proxy.decode()

res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')
for tr in soup.select("#proxy-table li.proxy > script"):
    proxy_id = re.findall(r"Proxy[^']+(.*)\'",tr.contents[0])[0]
    print(decode_proxy(proxy_id))

First few results:

62.80.180.111:808068.183.221.156:38159189.201.134.13:8080178.60.201.44:8080128.199.79.15:8080139.59.78.193:8080103.148.216.5:80