Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

That is actually very simple, and you can even use a headless browser to execute javascript:

first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582

selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..



> first install Xvfb and pyvirtualdisplay

You really don't need xvfb anymore. Use xserver-xorg-video-dummy.


Yippie...that worked. Thanks a ton for sharing that code snippet. Am unfortunately more of an analyst. Have only just started picking up coding for data analysis. JS scraping was one area that I always had difficulty with. Not anymore :D




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: