Comments on: How To: Use Python to Collect Data from the Web – Part 1: Parsing Data

By: Drew Conway

Drew Conway — Tue, 01 Dec 2009 14:32:04 +0000

Since this post is relatively old the likely explanation is that NFL.com has changed its web formatting. This will cause the script to go looking for data where it no longer exists; hence, resulting in an IndexError.

I suggest working with the code as is, but inspecting the HTML to see how the new indexing works.

By: satyam mukherjee

satyam mukherjee — Tue, 01 Dec 2009 13:15:19 +0000

hi nice information. I am new to Python. I tried to run the code myself but landed up with a following error :

Traceback (most recent call last):
File “players-trial.py”, line 53, in ?
player_urls=get_player_profiles(players)
File “players-trial.py”, line 37, in get_player_profiles
search_url=”http://www.nfl.com/players/search?category=name&filter=”+names[0]+”+”+names[1]+”&playerType=current&team=3410″
IndexError: list index out of range

Can any one help me in this regard
thanks in advance

By: How Python can Turn the Internet into your Dataset: Part 1 | Computational Legal Studies

Wed, 01 Jul 2009 23:05:55 +0000

[...] over at Zero Intelligence Agents has gotten off to a great start with his first two tutorials on collecting and managing web data with Python. However, critics of such automated collection might argue that [...]

By: How to Use Python to Collect Data from the Web [From Drew Conway] | Computational Legal Studies

Wed, 01 Jul 2009 04:24:24 +0000

[...] wanted to highlight a couple of very interesting posts by Drew Conway of Zero Intelligence Agents. While not simple, the programming language python offers significant returns upon investment. [...]

By: Drew Conway

Drew Conway — Tue, 30 Jun 2009 21:35:48 +0000

Clem, chalk that up to tutorial-esque redundancy, thanks though, for being the police.

By: Clem Kadiddlehopper

Clem Kadiddlehopper — Tue, 30 Jun 2009 19:41:34 +0000

Was “comman-delimited spreadsheet” supposed to be a joke, or do you not know what csv stands for?

By: Drew Conway

Drew Conway — Tue, 30 Jun 2009 13:32:43 +0000

Paul,

Thanks, I had never seen YQL before, but it does seem like a very convenient way to parse HTML (as long as you have some familiarity with SQL commands).

There are so many ways to approach this problem, and it is fun to think about them all!

By: Paul Tarjan

Paul Tarjan — Tue, 30 Jun 2009 05:16:03 +0000

I usually use YQL (which in turn uses html tidy) to parse webpages. It gives and E4X interface over the website, which I find is really nice for data scraping.

For example :

http://yourock.paulisageek.com/yql/trueachievements.xml

use ‘http://yourock.paulisageek.com/yql/trueachievements.xml’; SELECT * FROM trueachievements WHERE url = ‘trueachievements.com/ptarjan.html’;

http://query.yahooapis.com/v1/public/yql?q=use%20‘http%3A%2F%2Fyourock.paulisageek.com%2Fyql%2Ftrueachievements.xml’%3B%20SELECT%20*%20FROM%20trueachievements%20WHERE%20url%20%3D%20′trueachievements.com%2Fptarjan.html’%3B&format=xml

By: Drew Conway

Drew Conway — Tue, 30 Jun 2009 01:35:23 +0000

andrix,

I considered doing this tutorial with scrapy, and I agree that it is immensely useful, but I thought it was not the right tool for this particular problem.

For tasks that require persistent scrapping, like downloading stock price data, etc., it would be perfect. For these “one-shot” downloads this combination of tools is better. Maybe I will do a scrapy tutorial in the future.

Thanks for the suggestion!

By: andrix

andrix — Tue, 30 Jun 2009 01:21:53 +0000

I've recommend you to use Scrapy. Scrapy is an excellent framework for make Screen Scraping. As the scrapy.org says Scrapy is a high level scraping and web crawling framework for writing spiders to crawl and parse web pages for all kinds of purposes, from information retrieval to monitoring or testing web sites. Try it, you will not regret!