RM Tips and Strategy

Challenges of rateshopping

James Godfrey
December 16, 2019
In order to determine the best price for your product given your current sales performance, it is important to have an idea of the current market rates and the direction that prices are moving.

In this article, WeYield summarizes the challenges of collecting competitive pricing data and the factors you should consider when reviewing data suppliers.

Why is automated rate scraping an important part of the pricing decision process ?

In order to determine the best price for your product given your current sales performance, it is important to have an idea of the current market rates and the direction that prices are moving. This will enable you to raise prices (and RPD) confident in the knowledge that it might not affect your overall positioning in the market. Alternatively, if you need to generate demand because you are behind schedule, you can lower prices gently to a more appropriate price level and potentially improve your position in results, without requiring a drastic overall drop (1).

Typically, small rental car businesses have collected this information manually and on an adhoc basis. However, this is not an effective use of your staff’s time and abilities. By automating the collection of this data, it frees up valuable time to spend analyzing the data rather than collecting it. It also ensures that the data can be collected in a consistent timely manner to help you identify key trends in the marketplace. Certainly, as your demand for more competitive information grows, you will soon reach the stage where the manual collection is not possible.

Cached vs. Scheduled & Real-time rate capture

There are many rate shopping suppliers in the marketplace offering the ability to collect competitor prices. It is important to establish if the supplier is really collecting prices on a real-time capture basis, or if they are accessing a cache of prices. At WeYield, we collect prices on an on-demand basis. The queries are scheduled to run during the night, so data is available to the clients in the morning of their business day. On top of this, we recently launched Rateshop planner. This tool allows clients to build and schedule their own rate collections, which is advantageous if you need to review destinations/durations/dates, not in your standard scheduled data collections.

Cached prices are not the same as real-time captures, which means that the data a client sees, is not captured on demand, but periodically by the rate shopping supplier (with no input from the client), and the cache/ temporary storage itself is queried when the client requests their data. This can speed up data retrieval, but the cached data may already be out of date. Often, if there are data collection issues, the gaps are plugged in by older cached data. Sometimes suppliers use an API to access the data used to fill the cache, this may not always match the prices displayed on the competitor live websites. There is potential that if using cached prices to make pricing decisions, you might not be making accurate judgments if the data itself does not reflect the price a real customer would see.

Scraping and bot detection

Prices are collected via scripts often referred to as bots. These are pieces of code that are built to visit a website, perform a search (as a customer would do), and then record the results. As the internet has evolved, this has resulted in websites becoming more dynamic (i.e they frequently change layouts, so scripts to scrape data have to be continuously rebuilt) and at the same time, websites have become better at recognizing visitors that are not human, and in turn blocking these or worse, misleading the ‘bot visitor’ with incorrect pricing and availability. To counter this data scraping companies use a variety of techniques to mask their identity.

  1. Firstly, they utilize IP addresses supplied by multiple IP suppliers, so that the target sites see visitors from different locations.
  2. Secondly, the script technologies have improved so that the site visit behavior better resembles human website browsing.
  3. Thirdly each data collection is done as a clean visit, i.e there is no cookie history attached to the IP, it’s as if a brand new computer is used, using a browser for the first time, with no other search history, hence the target site has no cookie information to identify the customer, This, in turn, limits the visits exposure to the future problem of personalized pricing.

Rise of personalized pricing

As the availability of customer data has increased, travel businesses are becoming increasingly clever in the way they can promote products to customers. In the airline business, for example, the New Distribution Capabilities (NDC) technologies, are starting to allow airlines to price a flight ticket differently to every search customer based on a number of factors, including but not limited to past booking history, reward membership status, time to departure, travel purpose, general demand for the flight. This will become the norm in the airline industry in the next few years, and will likely be the same on OTA and large Chain Hotel websites too. By doing a ‘clean’ search automated using the technology of WeYileld’s partners, our clients will not see these personalized pricing effects, just the general price intended for the product.

Why might prices differ via manual collection vs. automated means ?

Our clients often manual check the prices we have collected and it is quite possible there will be some variance. This is likely due to a number of factors:

  1. firstly, prices may have changed since our data collection time due to more automation in the marketplace,
  2. secondly, our price is from a ‘clean shop’, it is quite possible that your browser contains lots of browsing information/cookie history and this might influence the prices the target site displays, especially if it detects that you frequently search but don’t book anything. We have heard instances of major brand car rental companies, not being able to manually shop their competitors' sites from their offices as the IP address is recognized and blocked.

Other considerations: Day time throttling / GEO IP Impact

Data vendors have to be conscious of the activity of the sites they scrape, so as not to disrupt their core business (selling products to consumers) as a result we typically see slower data collections during daytime business hours. This is fair so as to not disrupt traffic to the sites and risk being blocked permanently from collecting data from them. Another consideration is the impact of different IP addresses on the prices and availability of products captured during scraping. Clever brokers and brands now utilize technologies to show different prices to different customer markets. It is important that your data vendor is aware of which sites perform this activity and are able to force the correct use of geographical IP addresses when scraping as appropriate to the data request.

Future challenges – Member-only rates, Increased appetite for data /automation of price setting.

In the future, we are seeing some travel businesses push priority pricing, for customers who have logged into their site or are part of their rewards scheme. It is important that data vendors monitor this closely as it develops so that you continue to get an accurate picture of pricing in your market. We are also seeing an increased appetite for competitive data as tools are developed to help in the automation of pricing, WeYield is itself looking into this area with great interest.

To summarize:

  • Rate shopping via screenscraping has been around since the early 2000s and is utilized by key players throughout the Travel Industry.
  • Price monitoring is a key part of the pricing decision process when considered alongside up-to-date data about your own performance.
  • Insist on live shopped rates and not cached ones.
  • Ensure that the rate shopping supplier you choose has a history of high accuracy and the ability to navigate difficult sites / blocking technologies such as captcha pages.
  • Consider your supplier’s ability to scale as your desire for more competitive data increases, do they have a decent-sized IT team that is required to ensure all the web bots you use are maintained and accessible at all times.
  • Ensure your data supplier acts ethically when it captures data, you do not want to be blocked from collecting data from your key broker/sales channels during peak season!

Please do not hesitate to get in touch with WeYield if you have any further queries regarding the pricing data collection process.

  1. Refer to WeYield academy about top-down pricing. To be avoided at all times.


  1. https://hbr.org/2017/10/how-retailers-use-personalized-prices-to-test-what-youre-willing-to-pay

Published by
Linkedin logo
James Godfrey
December 16, 2019

I am a Customer Success Manager at WeYield, my role is to support Customers to achieve their goals. I like talking to customers to understand their needs and how they currently work. We push them to understand there are other ways to do things and challenge them to look at their own data in different ways.

Our readers also liked



Subscribe to our newsletter and be the first to receive the most relevant information and best practices on revenue management.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.