Get inner div using Beautifulsoup : Stepwise scrapping Implementattion

We can get inner div using Beautifulsoup by using find or findall() function from the Beautifulsoup library. There are two possible ways to achieve this. In the first option, we can directly search the class name using find() function in div components. The other one is using findAll() function from Beautifulsoup exactly similar to the earlier option of find() function with one very impacting difference of iterating all the div classes to custom control the logic in searching. This findAll() provides the flexibility to integrate the business logic like if you want to extract the div class of the same name in a different hierarchy etc. In this article, we will discuss all these terms in more detail.

Get inner div using Beautifulsoup : ( Implementation ) –

Let’s create a dummy HTML Content to make this topic easy and explainable. Here we will extract some inner dev.

Solution 1 : Using find() method –

If you want to scrap any specific tag from the HTML file. Let’s have the above HTML String and there we have ‘class5’ where we have the text “Data Science Learner” which we need to extract. In the below code, we will use the ‘BeautifulSoup‘ library and find() function. Here is the below code.


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
div= soup.find("div", class_="class5")
div.text

Get inner div using Beautifulsoup using find()

How to Use Beautifulsoup to parse html (html.parser)

Solution 2 : Using findAll() method –

findAll() function works identically similarly as find() function with the difference it extracts all the tags in list format. Here we can apply any loop like for etc to iterate and apply the business logic for extraction and scapping.


from bs4 import BeautifulSoup
content = BeautifulSoup(html)
output=''
for div in content.findAll('div', attrs={'class':'class5'}):
   output=div.text
print(output)