Discover the most innovative Selenium projects with code examples that will revolutionize automated testing in 2024. Discover the top 10 automation projects providing reliable software development, improving productivity, and fostering innovation in web testing. This blog examines the crucial technologies that are reshaping the testing scene from cutting-edge frameworks to advanced plugins. By getting involved with these influential Selenium projects that address various testing requirements and technologies, you can stay ahead of the curve.
Define Selenium
Selenium is a browser automation library you can use to automate some processes. It is a web automation library used to automate processes in a web browser such as logging in to a website, web scraping for data mining, and automating searching prompts.
Selenium is one of the most known skills on demand since it ties to the ability of data mining and data analysis. You can use the Selenium WebDriver to scrape through a website and mine all data, instead of manually inputting data and putting them in an Excel file. This is one of the reasons why if someone plans to go into the field of Data Science, they should learn how to use Selenium since it is an essential tool for Data Mining. Learn more on Selenium with Python with us!
There are a few advantages of using Selenium for your projects. They are:
Low cost: Selenium is an open-source software which means, anyone can use it for free with little to no cost.
Less time: Automating browser processes will save more time compared to manual processes, and it is more accurate too.
Uses of Selenium
Selenium can automate mundane tasks, auto-fill forms, open and close tabs, and check for links automatically. But, one of its most important uses is in web-scraping for data mining from different websites. This is used to reduce the time taken to mine data from different resources manually. These are the main positives of using Selenium.
Installation
As a browser automation library, Selenium is compatible with multiple languages such as Python, Java, C# and so on, where you can import the library Selenium into the code and start using it. Here, we’ll be covering how to use Selenium with Python here:
To download Selenium in your system, check if you have Python downloaded in your system. To do this, simply type the following command in your command prompt.
Common Prompt
python --version
After checking the Python version, we need to install the Selenium library in the system. The Command Prompt code is shown below.
Common Prompt
pip install selenium
After installing the Selenium library we can use it for web automation. But, before that, let us go through some prerequisites.
Requirements for the system before using selenium
To use Selenium web browser automation for your browser, it is important to download the web drivers for your respective main browsers that you want to automate. The links to download the drivers are given below.
Microsoft Edge: Click Here
Google Chrome: Click Here
Note: Only the web drivers are downloaded differently for each browser. After that, The coding part stays the same for all.
In this project, we’ll be using Microsoft Edge as the main browser for automation. Let’s see more about browser drivers.
Browser Drivers
Browser drivers are tools which provide authorization for the Selenium web automation library to work and allow web scraping in their browsers. The main browsers used for web scraping using Selenium are:
- Google Chrome: Chromedriver
- Microsoft Edge: Edgedriver
- Brave: Chromedriver
Google Chrome and Microsoft Edge provide web drivers you can use to automate web processes. The Brave browser is powered by Chromium from Google Chrome, hence, we can use Chromedriver for web automation in Brave.
Important: Once you’ve downloaded the browser web-drivers it is important to accurately configure it to an accessible path, preferably one where you don’t need to have administrative privileges.
Browser Elements
To interact with web pages, Selenium identifies and manipulates different HTML elements like buttons, forms, and input fields. You can inspect browser elements using developer tools in your browser. Here’s a quick guide on how to interact with elements:
- Right-click on the web page and select “Inspect” to open the developer tools.
- Use the Element Inspector to highlight and locate HTML elements, like buttons, forms, or links, that you want to interact with using Selenium.
Use Selenium’s methods like find_element_by_id, find_element_by_class_name, or find_element_by_xpath to locate these elements for automation.
Selenium Projects Using Python
Here are 10 selenium projects you can use by selecting HTML elements as shown above:
Example -1: Browser Automation with Google
In this example, we can automate opening Google and inserting in queries to find the answer and print the resultant. Here, we’re seeing how to find the Independence days of four countries:
In Selenium, web automation is done by selecting browser elements in a website running in a browser. For example, let us go to the website Google and see its HTML elements.
To do so, we can click left click and choose the inspect option or, we can press Ctrl+Shift+C and we will see the HTML elements running Google.
As we can see, we can view the HTML and CSS running the site. And, if we hover over the website elements, we can see it highlights the HTML class responsible for running that specific part.
With all this, let us see through some short example projects below.
- USA
- India
- Sri Lanka
- Pakistan
Code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.edge.service import Service
countries = ['US', 'India', 'Sri Lanka', 'Pakistan']
# Provide the correct path to msedgedriver.exe
s = Service("C:\\Program Files\\msedgedriver.exe")
browser = webdriver.Edge(service=s)
for country in countries:
try:
browser.get('https://www.google.com/')
query = 'When is independence day for ' + country + '?'
# Locate the search input field by name 'q'
text_field = browser.find_element(by=By.NAME, value='q')
text_field.clear() # Clear any existing text in the field
text_field.send_keys(query)
text_field.submit()
time.sleep(2) # Wait for the results to load
result = browser.find_element(by=By.XPATH, value="//div[@class='BNeawe iBp4i AP7Wnd']")
print(f"Independence day for {country}: {result.text}")
time.sleep(2) # Pause before the next iteration
except Exception as e:
print(f"An error occurred for {country}: {str(e)}")
# Close the browser when done
browser.quit()
In this code, by referencing the HTML element of the Google Search bar, we can input the Independence days to be printed. If we run the code, an instance of Microsoft Edge will be opened and we can print the answers by referencing the HTML class it is a part of. It comes along with a header, “Microsoft Edge is being automated by a test software”.
Browser Elements
The Output in Command Prompt:
DevTools listening on ws://127.0.0.1:55137/devtools/browser/55f1327d-1b9c-4b45-8d42-372c56f022a0
Tue, 4 Jul, 2023
Tue, 15 Aug, 2023
Sun, 4 Feb, 2024
Mon, 14 Aug, 2023
Example -2: Copy Page Details into another page
Let us see how we can copy the text of a website in Microsoft Edge, and how we can print its contents using Selenium. Here, we open a website using Selenium and copy its contents into a file, then show it.
Code:
from selenium import webdriver import time from selenium.webdriver.edge.service import Service countries = ['US' , 'India' , 'Sri Lanka' , 'Pakistan'] # Set the path to the msedgedriver.exe driver_path = "C:\\Program Files\\msedgedriver.exe" try: # Initialize the Edge browser s = Service(driver_path) browser = webdriver.Edge(service=s) # Navigate to the URL browser.get('"https://www.btreesystems.com/selenium-with-python-training-in-chennai/"') # Wait for the page to load (you can use WebDriverWait for more robust waiting) time.sleep(2) # Get the page source and encode it as UTF-8 page_source = browser.page_source.encode('utf-8') # Use a with statement to handle file operations with open('result.html', 'wb') as f: f.write(page_source) except Exception as e: print(f"An error occurred: {str(e)}") finally: # Make sure to close the browser, even if an exception occurs browser.quit()
This code creates a file called “result.html” by copying all HTML text of the given website. This is saved in the folder you’re working in.
Here, we’re utilizing the file-handling concept to encode it to ‘utf-8’, then we open the file of the website in HTML and write the HTML code in the file, hence creating the result.html file.
OUTPUT:
Selenium Output
The HTML code of the website is shown below:
Raw Code Selenium
Example -3: Check whether a Given Word exists or Not
This project idea deals with the fact that you can check whether a word exists on the website or not. It returns “True” if it does, “False” if it doesn’t exist.
Code:
from selenium import webdriver import time from selenium.webdriver.edge.service import Service s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) browser.get('https://www.btreesystems.com/selenium-with-python-training-in-chennai/') get_source=browser.page_source search_text="Selenium" print(search_text in get_source)
OUTPUT:
True
Example -4: Find the number of links present in a Website
When you want to find the total number of links embedded in a given website, you can use Selenium to find the HTML tags to find the number of links by finding the HTML tag “a”. Generally, in HTML, the “href” tag is used to embed links and “a” is the object used to embed links.
Code:
from selenium import webdriver from selenium.webdriver.edge.service import Service from selenium.webdriver.common.by import By s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) browser.get('https://www.btreesystems.com/') link=browser.find_elements(By.TAG_NAME, 'a') print(len(link))
After finding the links, we print the number of links present on the website.
OUTPUT:
124 The number of links embedded in the website is 124.
Example -5: Opening and Closing Tabs in the Browser
This project shows how to automate opening, redirecting to a tab, and closing tabs. Let us see how to do so below:
Code:
from selenium import webdriver from selenium.webdriver.edge.service import Service import time s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) url="https://www.google.com/" url="https://www.btreesystems.com/" browser.get(url1) browser.execute_script("window.open('');") browser.switch_to.window(browser.window_handles[1]) browser.get(url) time.sleep(2) browser.close() browser.switch_to.window(browser.window_handles[0])
After opening the browser, for the first link, we automate opening another tab and switch to the new window using the switch_to.window() function. To that tab, we redirect it to the second website. We then close the first tab, then switch to the current tab.
switch_to.window() function
Here, two tabs are opened, and after 2 seconds, the Google tab is closed and we switch over to the first website.
Opening and Closing Tabs in the Browser
Example -6: Automate Scrolling on a Website
This project shows how to automate scrolling in a web browser in Python using Selenium.
Code:
from selenium import webdriver import time from selenium.webdriver.edge.service import Service s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) driver.get("https://www.countries-ofthe-world.com/flags-of-the-world.html") driver.execute_script("window.scrollBy(0,2000)","") time.sleep(2)
This opens the flags of the website and scrolls them by 2000 pixels. The output is shown below: www.countries-ofthe-world.com/flags-of-the-world.html
Automate Scrolling on a Website
Example -7: Automate Tests
Other than browser automation, we can use Selenium to automate tests for a given search to a website. Here, we’re searching on the python.org website to test using geckodriver or the unittest library in Python.
This library has some use cases to test the Selenium automation and tests based on the predefined unit test cases in the library.
Code:
from selenium import webdriver from selenium.webdriver.edge.service import Service from selenium.webdriver.common.by import By import unittest class PythonOrgSearch (unittest.TestCase): def setUp (self): self.s = Service("C:\Program Files\msedgedriver.exe") self.driver = webdriver.Edge(service=self.s) def test_search_in_python_org (self): driver = self.driver driver.get("http://www.python.org") self.assertIn("Python", driver.title) elem = driver.find_element(by=By.XPATH, value='//*[@id="id-search-field"]') elem.send_keys("pycon") if __name__ =="__main__": unittest.main()
Here, we define a class to run the tests on and send keys to the search bar. This will return “No results” for this example and the testing will be successful.
OUTPUT:
——————————————————————————
Ran a test in 6.538s Ok
Example -8: Automate Form Filling
Do you find it annoying to fill out Google Forms all the time? Now, it is possible to automate filling out the form using the Selenium library in Python. By referring to the HTML elements used for input in Google Forms and sending prompts, we can automate filling forms in Google and submitting them.
Code:
from selenium import webdriver from selenium.webdriver.common.by import By import time from selenium.webdriver.edge.serviceimport Service s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) driver.get("https://docs.google.com/forms/d/e/1FAIpQLSezs9vfDnGxHrXsF52bxKXdrmlQHbHI0HsxrO6A9PGbC3K0Xw/viewform?usp=sf_link") time.sleep(2) Name="Prabhu Deva" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(Name) age="25" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(age) email="prabhudeva76@gmail.com" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(email) email="India" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(email) email="Chennai" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(email) email="Teynampet" n=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') n.send_keys(email) submit=driver.find_element(by=By.XPATH, value'//*[@id="mG61Hd"]/div[2]/div/div[2]/div[1]/div/div/div[2]/div/div[1]/div/div[1]/input') submit.click()
By finding the HTML element by hovering over it, we can copy the XPATH code of the Input variable to automate form filling
XPATH
To copy the XPATH code, simply right-click the code and select Copy as XPATH.
Copy as XPATH
After doing so, we can input in the XPATH our code to send prompts to the Google form, including the submit button which we can click using the click() function in Selenium. This results in the output shown below:
Form Testing Submit Form
Example -9: Browser Automation For Google Maps
This innovative idea performs web automation of the browser, where we open Google Maps, and enter our location, which automatically shows us the map location and the directions to travel to it.
g_link="https://www.google.com/maps/@13.0478078,80.0442021,11z?entry=ttu" ip_link='//*[@id="searchboxinput"]' dire='//*[@id="hArJGc"]/div'
import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.edge.service import Service s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) browser.get(g_link) time.sleep(2) name=str(input("Enter Location: ")) n=browser.find_element(by=By.XPATH, value=ip_link) n.send_keys(name) directions=browser.find_element(by=By.XPATH, value=dire) directions.click()
By HTML inspect element, we find the input box to send our input prompt and also the direction icon to show us the travel time which we can make Selenium web automation software click automatically without any manual prompt. The output is shown below:
OUTPUT:
Enter location: Bangalore
Map Testing Thus, we have successfully automated Google Maps using Selenium.
Example -10: Sign in Automatically on LinkedIn
This project makes use of signing in websites and automating them using Selenium. Let’s see how to do that below:
Code:
import time from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.edge.service import Service s = Service("C:\Program Files\msedgedriver.exe") browser = webdriver.Edge(service=s) browser.get("https://www.linkedin.com") username=browser.find_element(By.XPATH, '//*[@id="session_key"]') username.send_keys("EMAIL") #Enter Your Email username=browser.find_element(By.XPATH, '//*[@id="session_password"]') username.send_keys("PASSWORD") #Enter Your Password button=browser.find_element(By.XPATH, '//*[@id="main-content"]/section[1]/div/div/form/div[2]/button') time.sleep(2) button.click()
Here, we start by opening the LinkedIn page which will ask you to Sign In. Previously, we learned how to copy XPATH variables for the input boxes to input in your username and password and also to click the button “Sign In”. This will automatically input the username and password you’ve put in the code. The output is shown below:
OUTPUT:
Linkedin Login
Here, we can see that we’ve successfully logged in to LinkedIn.
NOTE: Some websites like Instagram and even LinkedIn sometimes block Selenium requests due to growing concerns about data privacy. Hence, the code may not work if you run it one too many times.
Syntax Updates
As a result of technological advancements, all websites including the browser go through updates due to which there is a high chance that the Python syntax for the Selenium library may change in the future, either deprecating the previous functions to encourage developers to accept the new syntax or rendering them obsolete.
For Example:
browser.find_element_by_xpath(value)
This syntax has been deprecated and now the new syntax is
from browser.find_element_by_xpath(value) import By browser.find_element(By.XPATH, 'value' )
These are small examples of syntax changes that have happened over time.
Future Scope of These Selenium Projects
These Selenium projects with code examples serve as a great introduction to the web automation library Selenium. Selenium has been instrumental as a base automation tool to perform web-scraping, browser automation, search automation and sending in requests. All of these play a huge role in automating the more mundane tasks that developers need to do daily. But, recently, due to many restrictions on data scraping on websites as a result of protecting user identity, many websites, like Twitter have prohibited the limit of scraping tweets and also made it difficult to log in using a browser automation tool due to the presence of different types of CAPTCHA texts to prevent bots or users with malicious intent to scrape sensitive data and sell it to the highest bidder.
As a result, to combat unwanted scraping, but also to make data mining easier, such websites have started providing APIs to assist in data mining easier without having to resort to web scraping.
Conclusion
From the above Selenium project ideas, we can see how versatile and loaded with functionalities on Selenium projects with code examples. Since it is a web automation library, websites are constantly changing their HTML codes while bringing new updates to their website. Despite its useful functionalities, its major drawback is that the syntax keeps constantly changing and many more websites introduce CAPTCHAs to prevent the growing data scraping problem. There is also a concern over the ethics of web scraping where the website owner may not have any idea that his website is being scraped for data, and now only some websites allow scraping by Selenium.
