Why TF*IDF Doesn't Allow for Content Optimization

Many SEO tools or consultants base their content creation and/or optimizations on the TF*IDF method. Although TF IDF gives us the impression of improving our content, it does not actually solve SEO problems.

By learning more about its use and operation, you will discover that the use of TF-IDF can mislead your content optimizations.

What is TF*IDF?

TF-IDF (for Term Frequency and Inverse Document Frequency) is a measure used to determine the relevance of a term in a document. The formula takes into account the frequency of a term (TF) in a given document as well as the number of documents containing this word (IDF). TF IDF helps to distinguish differentiating elements (here words) from one document to another.

To learn more about the method and its calculation, go here https://www.seoquantum.com/billet/optimisez-vos-contenus-mots-rares

Does Google use the TF IDF method? Is this measure still relevant?

Google (via John Mueller) hinted that the search engine's use of this method is limited. He mentions TF-IDF for the first time when he addresses the subject of excluding stop words.

This is not surprising given the advancement of the Knowledge Graph database and the Hummingbird and Rankbrain algorithms. Google is indeed constantly evolving. Its understanding of language is constantly improving as it learns to deal with the ambiguities of human language.

Google also improves its ability to handle queries with multiple meanings. Despite everything, the algorithm is far from perfect. As we will see, this poses a serious challenge to those who use the TF-IDF analysis method for content optimization.

Why does TF-IDF give us the impression of working?

Despite Google's limited use of this dated technology, many SEO consultants and semantic tools appreciate TF-IDF. Why?

TF-IDF is a relatively unknown concept within the SEO community. Because this analysis method is not familiar to them, many SEO experts or tools mistakenly think it is cutting-edge technology. This gives it a certain prestige.

Few know the history of TF-IDF. Most do not know its true age (the 1970s) or its true purpose. Hint: this method was not created for content optimization. To learn more, visit the works of G. Salton and K. Spärck Jones.

SEO experts believe that TF-IDF plays an important role in the operation of Google's search algorithms. Because several patents and a few publications refer to it, there is a mistaken assumption about the role this technology plays.

TF-IDF appears as a sophisticated method for most SEO consultants. It is rare that they have been trained in data science. That's why it's easy for them to assume that the apparent complexity of this method translates into its effectiveness.

Who wouldn't like to use a sophisticated and revolutionary technology for engine optimization? It sounds so promising!

Except that it's not.

6 difficulties encountered with TF IDF

There are a number of free or inexpensive SEO tools that promise to help you optimize your content using the TF-IDF analysis method. All these tools have the following problems.

TF-IDF is a primitive approach

TF-IDF allows you to measure the importance of a document within a corpus, based on a given term. Its skills are limited, especially when you use synonyms. Indeed, a document considered very relevant for "baby" may be ignored for the term "infant".

Google, on the other hand, knows that the words "baby" and "infant" are strongly related (they are synonyms). It understands that a page relevant for one is probably relevant for the other, unless there are context clues in the rest of the query that prove otherwise. This is based on co-occurrence as well as the probability that they are both used in similar contexts.

Using TF to determine the importance of a term is an imperfect measure

Determining the importance of a term based on its frequency of use in a SERP is an imperfect measure.

If the search intentions of half of the corpus differ from the other half, the weight of the term (its importance) will be 50%. However, if all the documents in this corpus use a common word, it will be considered as the most important term regardless of the intention.

So, you will have to choose and focus on a single intention. But the tool will discourage you, as only five results use the term. It will tell you that there are only five results out of 10.

The IDF, on the other hand, allows to counterbalance the TF measure to determine the rarity (the differentiating elements) of a page.

The use of the method relies on Google's SERPs

Semantic tools using TF-IDF generally exploit the first 10 or 20 results of a SERP without studying the reasons why these pages contain these topics, thus raising two biases:

1. Pages may owe their "good" positioning to factors other than content, such as link building for example
2. The use of a small number of documents significantly affects the quality of the results. These tools do not take into account mediocre quality content or short texts.

The margin of error is so high that even taking into account the weaknesses of these tools, you will not have the necessary information to make informed decisions.

I suggest you save time by using other more effective tools. It is important to analyze all the content that addresses your topic.

The TF-IDF analysis method and the tools that calculate keyword density do not allow this. If you follow their advice, you will have as much chance of success as if you had played the trifecta.

TF-IDF analyzes and groups pages with different objectives

Selecting all the pages appearing among the first results of Google creates other problems. You risk including pages that are too general, too specific, or related to a different industry than yours.

In addition, TF-IDF does not understand search intentions.

In other words, if you have quality content, focused on a different search intention, you will be misled.

If you have poor quality content whose off-site web referencing has been well optimized, you will also be led down the wrong path. If you are hesitating between several intentions, the tool will not be effective either.

In blue, pages with an informational objective, in green pages with a commercial objective, and in yellow a transactional objective.

Tools that use the TF-IDF method only take into account pages

By limiting themselves to pages, these tools are not aware of the entirety of your website.

Writing a single page on a subject is usually not enough to optimize content. To do it right, you will need to create other content that will increase your topical relevance and allow the use of anchor texts and internal links.

At SEOQuantum, we have created the semantic crawler to help you with this task.

A score that has no meaning

Giving a page a score based on its compliance with TF-IDF seems at first glance to be a good idea. But if you can't learn more about the website or the page, this information is meaningless and not actionable.

Consider that the page with the highest score may:

• have a different objective than yours
• Have much more or much less authority
• Have multiple objectives
• Cover multiple topics

We believe in AI and its valuable help in enriching content, especially with key concepts. Here for baby monitor, the AI has distinguished 3 concepts: the functions of the device, the emission of waves, and finally the distance of the transmitter.

Help, my copywriter uses TF IDF

Tools using the TF-IDF method encourage bad habits among copywriters and SEO experts. They try to build content around words that are not suitable or add sections that do not match the search intent.

Even though it is possible to find inspiration thanks to this list, it is far from being a real solution.

What happens when you create a list of keywords using this methodology? The topics and intentions of the different terms will vary. The person who receives this list will not know what to do with it. It's just inefficient.

TF-IDF: the advantages

Despite its inefficiency and inaccuracy, it seems that there is value in using this type of approach. This method allows among other things to inspire you or make you discover a subject you had not thought of. But also it can help you realize that you have over-optimized your page (too many keywords...).

Conclusion

Other resources:

• You can consult the article by Quentin Fily TF-IDF: Determine a relevance score. Be careful, I do not agree with his conclusion on TF IDF ;)