Lesson finished
Up next: Website Architecture, Page Layout and Keyword Research for SEO
Unlock all 17 lessons of Search Engine Optimization & Search Engine MarketingFull course ₹40,000 · paid
Why SEO Matters and How Search Engines Crawl, Index and Rank
A free lesson from Skill-Lync's Search Engine Optimization & Search Engine Marketing course, on Grow by Skill-Lync · Last updated 16 Nov 2026 · Session: “Week 1 Session 2”
What will you learn in this free Search Engine Optimization & Search Engine Marketing lesson?
The instructor uses examples such as finding a cheaper phone seller and a nearby kulfi store to show why SEO matters. The lesson then walks through crawling, indexing, processing, relevancy calculation and retrieval, and defines algorithm, crawling, indexing and PPC.
- Search intent and organic traffic
- Crawling by search engine bots
- Indexing and processing of content
- Relevancy scoring and personalised results
What comes after lesson 2?
The next free lesson is lesson 3, “Website Architecture, Page Layout and Keyword Research for SEO” (27 min). Later in Skill-Lync's Search Engine Optimization & Search Engine Marketing course (paid): lesson 4, Week 2 Session 2; lesson 5, Week 3 Session 1; lesson 6, Week 3 Session 2. The full course has 17 lessons (6 h 9 min of video); see the course overview or its price and syllabus.
Lesson transcript
Machine transcript of the lesson audio (lightly cleaned). Technical terms may contain errors; the video is the reference.
Welcome back everyone. So we are now in the session two for week one and we'll be talking about fundamentals of SEO. Now just to recap quickly about what we saw last in session one, we were seeing the difference between SEO and SEM. So basically SEO will give you organic traffic to the website and SEM will give you paid traffic to the website. Now a PPC campaign predominantly will be sponsored search results or paid search campaigns. So therefore it will be paid search traffic. Now moving on into the session two, we'll be talking about importance of SEO. Slide 10, importance of SEO. Nowadays, everything is done online. Okay. Thanks to the pandemic, we have become quite versatile in using the internet to get stuff done.
So everything is done. So everything is done online. Search engines are the main source of web traffic because not everybody knows or likes to go back to the same website. Let me explain that better. Suppose you bought something on Amazon. You liked the product that got delivered. You loved it and you're okay with that. But then when you were planning to buy another mobile on Amazon, you saw that it is available for a lower price than Flipkart. So what would you do? Obviously, you'll buy it in Flipkart because you're going to get the same 10 day return period, everything very similar, but the price is obviously lesser. So you'll be buying it in Flipkart.
Now that being said, if some of your friend is talking about buying a new mobile and then he's saying that he saw some other website that was cheaper, but he's not able to remember the website. So what will you do? You will end up searching for it, obviously. So the moment you start searching for it, that's when your search intent is clearly visible to Google. So for example, in this example, which I was talking about, you are trying to find out a website that sells mobile phones for cheaper and a very particular mobile model, which your friend was suggesting or talking about in our scenario. Let's keep it like Google Pixel 7. Okay. So Pixel 7 is coming up and your friend saw a website that was selling it cheaper than what it is listed in Amazon.
Now, mind you, Pixel 7 is not sold straight away by Google yet in India, and it is partnered with Flipkart to sell it. But US sellers have already started listing it in Amazon India and people are buying it as well. So obviously the price will be, you know, higher because of that. Yeah, your friend found another website that sells it for cheaper. So what you would do is, you know this, you will find out about this while searching. You will search for buy Google Pixel 7 and then Amazon link will come, but that's pricier. So you will try and understand where is it going to be. Then you will see news articles which talk about Google tying up with Flipkart to sell Pixel 7.
And then again, you will see more news articles which talk about buying products in the US via websites in India. So and eventually you will come across other websites which, you know, ship products from US to India. At a cost. And finally, you will end up finding that, you know, amazing website which your friend had found where Pixel 7 was available for cheaper price than Amazon. And in this scenario, for the sake of Amazon. And in this scenario, for the sake of an example, let it be DesertCart. So DesertCart is a service website that, you know, buys products in the US, you know, packages them and ship them to your address in India.
So you've actually found that that was the website where your friend saw the price to be lower than Amazon, right? All this information, gathering, processing and, you know, final solution of a website that buys products in the US and sells them to you in India was found by searching in the Internet. So SEO is very key for this. Now, imagine DesertCart didn't do SEO. You will never be able to find that website. This logic of buying products in the US and selling them or shipping them to India at a cheaper cost.
This whole logic was embedded by SEO professionals into DesertCart website's content. Otherwise, when you search for something like this, obviously their website won't come up. So SEO is a key element for online marketing, especially when you need visibility from relevant users and SEO mainly generates organic traffic, organic search engine traffic. So that's the short form organic traffic. Now, SEO increases website traffic without involving the need to pay search engines. Obviously, only when you're doing paid search campaigns, you would pay Google. If you're doing paid search campaigns on Bing, you'll pay Bing. If you're doing it on Yahoo, you'll pay Yahoo.
So SEO is the tactic which gets traffic organically from search engines without you needing to pay for it relevant traffic, mind you. So that is the importance of SEO. Now, I know this is quite simple. I've tried defining it in just five points. But basically, if you notice SEO is required for everything. Now, let me explain that you're trying to buy something fancy. Something fancy being a Kulfi ice cream. So you want a Kulfi and you also want some ice cream. So this is your requirement.
Again, you want to buy it and eat it right now. This is your requirement. So what would you do now? You will call up your friend and ask, hey, where is the Kulfi store, man? I don't know. I need to buy and eat a Kulfi. I want a Kulfi. No, you're not going to do that. Obviously, you'll not do that. You'll either go on to Swiggy or you'll go on to Zomato. You will see the store. That is one way of doing it. The other easiest way is you'll go on to Google and you'll search Kulfi stores near me.
That is going to be the search you'll be doing. And immediately Google will show a list of businesses on Google Maps which sell Kulfi. Now, if you do a search Kulfi and ice cream store near me, it will show a list of businesses which either sell Kulfi or ice cream or sell both of them together in the maps and in general the search listings. And it might also have Zomato and Swiggy links now. Google is smart. You know, they keep updating their algorithm. That's a word I'll be explaining as well.
So this is what you do. And what you're doing is you're solving your problem of eating a Kulfi right away by finding the nearest Kulfi store, going there and buying it and eating it. Now, that's just one example like this. If you notice, Internet has percolated into our life. Anything you want now is literally available on the Internet. You want to see your favorite movie, you can, you know, wait for some time, get it on any of the OTT platforms and watch it at home or watch it on your laptop or watch it on your mobile.
That's the power of Internet. It's literally changed the way we used to process things. And also there are, you know, advantages, disadvantages. But let's not get into that. Let's just talk about SEO. So SEO is the thing that is helping Swiggy and Zomato get a lot of traffic. You'd be surprised how much Swiggy and Zomato invest in SEO. They invest a lot and they are not dumb. They do a lot of paid advertising. Agreed. There are amazing push notifications from Zomato. Agreed. But at the same time, if you honestly look at things, even a push notification is part of your SEO strategy.
Every single thing which is done on a website or an app is part of the SEO strategy. So SEO is extremely important for a business to survive online because without SEO, you will not be able to retain traffic to your website. Without SEO, you will not be able to gain new users to the website. And therefore, that will be the end of things. So that's in a nutshell, the overall importance of SEO. So moving on. Slide 11 working of search engines. So in this, we'll try and attempt to understand how a search engine functions.
Slide 12. I know there's a lot of stuff in that image on the right hand side, but let's try and understand. So when we consider working of search engines, we have to basically see the different components that are involved in it. And the image on the right is anatomy of search engine. It talks about selection and crawling, build inverse index, index serving the matching funnel, and then the search engine front end. All these might seem like highly technical terms, but they're not. Let me explain them one by one.
So crawling. In crawling, what a search engine does. So everything with search engines is bots, robots. So in crawling, it will be a spider bot. Reason being internet is the web and spider bot is the spider. Jokes apart. What a spider bot does is it crawls. It literally goes through every single new website that comes online. That's the only task for it. You would think that's simple, right? No, it's not simple. It might have been difficult some 10 years back, like internet is just 30 years old, but it might have been difficult until 10 years back.
But since 10 years, we have had so many, you know, automated or self-serve websites wherein you can create a new website just like that. You can go to Wix.com, you can go to Squarespace.com and, you know, just like that, build a new website for yourself and make it live. So, making a new website is quite simple now and what that implies is there are lots of new websites that come online on a daily basis. And a crawling bot, a crawler from Google search engine will go through all these new websites.
Mind you, it will not just go through all the new websites, it will go through every single website on the internet which is publicly available and it will try and crawl and understand information differences. Like, for example, November 1st, your website was crawled by Googlebot. November 3rd, your website is crawled again by Googlebot. But this time around, Googlebot will try and understand if there is fresh content added or if there is a deletion of content that's happened or is there more validation of content that has happened, more links that have been added.
It will try and understand all this stuff. And that's exactly what Google does. So, I'll talk about that. It is not part of crawling, but the crawler bots will do for both new and old websites. And there is always a period of frequency set for every single website by Googlebot to decide when they can crawl what frequency. Now, that frequency, they try and map it to the individual website. So, if a website on an average is publishing new pages every three days once, they will try and crawl that website every three days once.
And again, I'm saying try and crawl because the sheer number of websites, there are billions of websites. Crawling them, going through all of them is crazy. Imagine how you would feel if you had to read an 800 pages worth book in one hour. Painful, right? Now, imagine reading it three days later to see if there was a difference in it and find out the difference and add that to your brain. Impossible, right? But that's what crawlers do. They crawl, they try and identify differences and report it back.
So, the crawl. The second point is indexing. Now, what is indexing? Indexing is, you know, tagging content and categorizing it or cataloging it. Easiest way to understand this is if you have an online store. So, you have different products. Each product can come under a different category. And each category can come under a broad, bigger category. Bigger category being electronics. Broader category being mobiles, TVs or recorders. And then subcategories being mobile variants, mobile accessories, mobile softwares, etc. So, all these things can be cataloged, right? They can be having tags, right?
Everything can be tagged into something relevant or grouped into it. Grouping is a better word. Yeah. You can group things, right? That's exactly what indexing is all about. It tries to catalog your content. It tries to make sense of your content and file it accordingly. Okay. Processing is that part which tries to build that index and maintains the index. So, as I said, crawler will identify changes in your content, right? If you have more fresh content, it will let the indexed section know. Hey, there is more new content that's come.
Find out what it is. So, processing will try and adjust all these new content or content delusions. It will try and validate. Processing also tries to validate your authority. Remember, I told you at the start of week one in session one about authority. Yes. So, processing also takes into account authority and the quality of content. So, it's very important that you have extremely good quality of content and very good authority of the content. So, how would you build authority? By selling good stuff. By keeping good content.
By keeping extremely good images. That's how you build quality. Like, that's exactly how a physical store is maintained. Keeping your store clean, checking their expiry days and keeping the right products in the right catalog or in the right column or in the right shelf is important for a physical store. Same way, processing does the same thing. It maintains the overall index that Google has built. Okay. And then you have two key points, which is calculating relevancy and retrieving. Now, retrieving can also be seen as serving.
So, calculating relevancy is also called as matching funnel. Those are all bigger terms. But, simply put, what it means is it tries to match to the user query in the search engine results. Let's go back to our first example, which is buying leather shoes. So, if this is your search query, if this is what you put in Google search, then what Google's engine will do is it will try and go back to the index. It will try and see what all things are relevant for this particular search query, which is buying leather shoes.
Then, it will create a relevancy matrix. Basically, it will give us relevancy score. Okay. For all pages. Based on the scoring, it will index them or there is a priority order created. Based on the priority order, you would be getting it in your search engine results page. This, publishing it in the search engine results page, can be called as retrieving or serving the content. Right? Now, Google, if you treat it like a single website, what it is trying to do is, it is trying to help you find things inside its own website.
And then, it will display what you are trying to find. Got it? So, it's very similar. So, let's see it in very simpler ways. You have a laptop bag and you're carrying your laptop, you're carrying some books, you're carrying a notebook, you're carrying a diary in it. Now, if your friend comes and asks for a diary, you will try to remember in which compartment of your bag did you put your diary. You will try to go there. You will pick the diary and give it to your friend.
Something very similar happens with Google search as well. It will try to match the relevancy, then it will go to the exact section, bring out the pages, rank them, and then show them to us. This is how a search engine functions. It crawls the internet, it indexes the entire content, it processes the content, tagging them, cataloging them, properly marking what each content means. Then, based on user search queries, it calculates relevancy scores, and finally, it gives you the search engine results. So, this is how a search engine functions.
I hope I made it simple for you guys, but yeah, hopefully, it was easier. Slide 13, SEO glossary. Slide 14, SEO glossary. Algorithms. Now, I didn't mention that I'll be talking about an algorithm, right? Like, I'll be explaining what an algorithm is. Now, just by definition, it means a set of rules or a procedure or a step-by-step instruction manual on how to do a thing. Basically, it is a logic of how you can do something. So, if you had to make a cake, then you have a recipe, right?
And in the recipe, you will have your instructions. Now, imagine you trying to just understand what are the ingredients and what are the steps just at a glance, and you writing down the overall process in a small paragraph. That is your algorithm for making a cake. It is not in-depth thing, but at the same time, it is the logic behind how things can be done. So, a search engine algorithm is the logic behind how to rank a website, how to rate it. Like, first, if you have to rank it, you should first know about the website.
So, how you can find new websites, how you can understand what the website is about, then how you can rank it, and then give it out to people based on their searches. So, this is what an algorithm is all about. Now, I am sure you would have heard about Facebook algorithm, Instagram algorithm, TikTok algorithm, and all those things. But yeah, even those things are logics, basically. So, Instagram's algorithm, let me try and explain that. Instagram's algorithm, show this guy content from his friends, okay? Then for some other person, show this guy content from shoe brands, because he likes content which is talking about shoe brands, or show this guy content about travel, because this guy is searching for travel or spending more time on travel-oriented content.
That's exactly how your reels get optimized. The more you see, the more better the reels get. Basically, it is not the real content that is getting better. It is the algorithm that is matching content which syncs with your mindset. So, if you notice more of travel-oriented reels, then obviously the Instagram algorithm will push out more reels which are travel-oriented. So, Instagram is also trying to categorize reels based on the content theme they are talking about, and then based on the content theme they are talking about, and then based on the Users who are consuming similar content, they push out that reel.
So if you see more of travel, they'll push out more travel reels. If you see more of breaking news kind of reels, then it will push out more of breaking news kind of reels. If you consume more of trending music kind of reels, then it will push out trending music kind of reels to you. That's exactly how an algorithm is. It gives you the logic behind execution of a task. But that does not mean that algorithm is a thing that is executing. It is the code that is executing, obviously.
But yeah, algorithm is a thing that is defining how the code should execute. Crawling. So crawling, the method by which search engines find web pages. Search engines usually manage web crawlers using their own algorithms. The algorithm will instruct the web caller to find relevant information in the internet after receiving a search query. Now, this used to be the scenario of the old. When Google was new in the 1990s. In the 1990s, when Google was its fledgling state, that's when crawler used to start crawling and then come back.
Then indexing happening and then coming and search results coming. Now, Google does not do all that. Google is already crawling. Google is ever crawling because, as I said, the sheer number of new websites coming online every single minute is quite high compared to 20 years back or 10 years back. So, Google can't afford to wait for a user to come and give the search query to start crawling. Google is always crawling. But yes, every single day, 15% of searches that come on Google are new. So, what happens is, the crawling algorithm keeps refining the crawlers to find more of this content.
So, for example, if everybody is searching more on NFTs, if that is an exploding search term, then the crawler algorithm will define that. Hey, crawlers or this many number of crawlers find out more content that is very specific towards NFT. Try and find that out. That is exactly what this crawler would be doing. That's what crawling is all about. Moving on. Slide 15. Indexing and processing. Okay. Indexing and processing. Very simple. The process of storing and organizing content discovered by crawling. So, whatever crawled, you store it and catalog it.
Very simple. Processing. Processes a search request by comparing the search query from the search engine with the indexed pages in the database. As I said, this is just one part of it. The other part is, it keeps adjusting your index as well. Slide 16. Calculating relevancy. Now, using the search query, the search engine determines how relevant each page in its index is. It is each page and there are millions of pages. So, yeah, it does that very fast. So, the calculation happens quite fast. There is a big AI engine.
There is a big machine learning engine as well, which keeps improving the speed of it or which makes it more relevant for each account. So, if you are logged into your Google account and if you are doing searches, then the search results which you get and the search results which I get for the same query will be different. It is because the AI engine and the machine learning, the algorithm for calculating relevancy is also continuously monitoring on a user level. Therefore, for each user, the search results are personalized.
They are relevant to you as a user and that's how the relevancy calculation happens. Retrieving is a final task carried out by search engines in retrieving the results. A browser window shows related web pages. Okay. PPC. PPC stands for pay-per-click. It is a type in advertising. Basically, search engine ads or search ads. Simpler. That's all PPC is about. So, slide 17. Summary. Okay. So, in this week with both the sessions, session 1 and session 2, we learned the following. We saw the overview of SEO, distinction between SEO and SEM, the working of search engines, the purpose and importance of SEO.
So, this is what we saw. Right? I hope this session was useful and this week was extremely useful. I really believe that you have a better understanding now about how search engines work, why SEO is important in general and what more you can do if you get free traffic. Please be assured, next week will be even more exciting and we'll be talking about more important things. Thank you for now. Have a good day.