Pages

2014-03-23

Crawling with Python 3 asyncio

I added some code to crawl.py in the example code of aiohttp(http client/server for asyncio) so that it crawls ODP and saves scraped data into SQLite database.
I'm not sure if I'm handling SQLite correctly, but it seems to work.
Python 3, asyncio(New in Python 3.4. PyPI), aiohttp and lxml are required.