Table of Contents
show
Crawling pages on the web has become an everyday affair for most enterprises. Too often do we come across offline businesses as well who’d like data gathered from the web for internal analyses. All this eventually to serve customers faster and better. At times, when the crawl job is high-end cum high-scale, businesses also consider DaaS providers to supplement their efforts. Learn all about web scraping AJAX pages in this article.
However, the web landscape too has evolved with newer technologies that provide fancy experiences to web users. Scraping AJAX pages elements are one such common aid that leave even the DaaS providers perplexed. They come in various forms from a user’s point of view-
1. Load more results on the same page
2. Filter results based on various selection criteria
3. Submit forms, etc.
When crawling a non-AJAX page, simple GET requests do the job. However, scraping AJAX pages work with POST requests that are not easy to trace for a normal bot.
 |
| GET vs. POST |
At PromptCloud, from our experience with a number of AJAX sites on the web, we’ve crossed the tech barrier. Below is a quick review about the challenges that come with AJAX crawling and its indicative solutions
How do I scrape a website using AJAX?
1. Javascript Emulations–
A bot essentially emulates human browsing to fetch pages. When this needs to be done for Javascript components on a page, it gets tricky. Headless browser, which emulates human interaction with a web page without an interface, is the current approach. These browsers click on various elements/ dropdown lists that are embedded within Javascript code and capture responses to be transferred to programs. Which headless browser to pick depends on what fits well into your current stack. For users wondering is JavaScript good for web scraping, we’ve got you covered.
2. Fetch Bandwidths–
Unlike GET requests which complete pretty quickly, POST requests take quite a bit of time due to the number of events involved per fetch. Hence a good amount of bandwidth needs to be allocated in order to receive the response. For the same reason, wait times need to be taken care of too else you might end up with incomplete responses.
3. .NET Architectures–
This is a more complex scenario related to maintaining the View State. Most of the postbacks come with an event and its validation. The bot needs to track the view state and pass validations for the event to occur so that the code can be executed and results captured. This is achieved by adopting a mechanism to restore states if things break midway.
4. Page Encoding–
Request and response headers need to be taken care of on AJAX pages. The request needs to be sent in the exact format as expected by the server (Content-type or media type, accept fields, etc.) and similarly responses need to be parsed based on the content-type.
A Use Case
One of our clients who is into sale of event tickets at discounted rates had us crawl one of the ticketing sites on the web weekly; one of the most complex AJAX crawling we’ve dealt with so far. For the data that was to be extracted, multiple AJAX fetches were needed depending on the selections made. Requests had to be made for a combination of items from the dropdown box. These came with cookies and session IDs. To add to the challenge the site was extremely dynamic and changed its structure every week making it difficult for us to follow what data was where on the page.
We developed an AJAX crawler specific to this site to take care of all the dynamics. Response times were taken care of so that we didn’t miss any relevant information. We included an ML component to improve the crawler which is now pretty stable irrespective of changes on the site.
Overall, AJAX crawling requires more compute power in addition to the tech expertise. And because there’s no uniformity on the web, there’s always a new challenge to overcome in this landscape. It wouldn’t be an overrating if we said we’ve done a good job at that so far and have developed the knack 🙂
Scraping AJAX pages with python can be done if you have some coding knowledge. If you are looking for large scale scraping, we can help. Reach out to us for any kind of web scraping/ crawling- either AJAX or not. Also, get in touch to understand is scraping pages legal or not. We’ll take care of the complexities.
