Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add agents that they can spider dynamic content #1

Open
kltm opened this issue Jul 13, 2015 · 1 comment
Open

Add agents that they can spider dynamic content #1

kltm opened this issue Jul 13, 2015 · 1 comment

Comments

@kltm
Copy link
Owner

kltm commented Jul 13, 2015

Currently, tanuki can only interact with static content. An agent should be added that can access dynamic JS content (maybe waiting a few seconds before processing or waiting for connection quiescence).

Since there don't seem to be many native methods for this, possibilities include:

  • wait for a native implementation of something (maybe webapi)
  • get the stack to run on abcl, then use selenium/phanomjs bindings more-or-less directly
  • pass through to an intermediate agent that can deal with this
  • compile down the code via parenscript to run phantomjs/etc natively (would require a better abstraction layer over the db)
@kltm
Copy link
Owner Author

kltm commented Jul 14, 2015

Actually, there may be a way through this with Electron as well?
https://news.ycombinator.com/item?id=9877314

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant