10
I need to do some web scraping. After playing around with different web testing framework, of which most where either too slow (Selenium) or too buggy for my needs (env.js), I decided that zombie.js looks most promising, as it uses a solid set of libraries for HTML parsing and DOM manipulation. However, it seems to me like it doesn't even support basic event-based Javascript code like in the following web page:
<html>
<head>
<title>test</title>
<script type="text/javascript">
console.log("test script executing...");
console.log("registering callback for event DOMContentLoaded on " + document);
document.addEventListener('DOMContentLoaded', function(){
console.log("DOMContentLoaded triggered");
}, false);
function loaded() {
console.log("onload triggered");
}
</script>
</head>
<body onload="loaded();">
<h1>Test</h1>
</body>
</html>
I then decided to trigger those events manually like this:
zombie = require("zombie");
zombie.visit("http://localhost:4567/", { debug: true }, function (err, browser, status) {
doc = browser.document;
console.log("firing DOMContentLoaded on " + doc);
browser.fire("DOMContentLoaded", doc, function (err, browser, status) {
body = browser.querySelector("body");
console.log("firing load on " + body);
browser.fire("load", body, function (err, browser, status) {
console.log(browser.html());
});
});
});
Which works for this particular test page. My problem is a more general one, though: I want to be able to scrape more complex, AJAX-based sites like a friends list on Facebook (something like http://www.facebook.com/profile.ph