Use Wookie to browse Web in Scala/Java

Introduction

In this article I will show you how to:

• Google Search with Wookie
• Login into GitHub and star a project
• Download JDK with Wookie 
• Use extended/a little more advanced API

The purpose of Wookie is to minimize the effort of the developer when using WebView. You can a compare how easier it is to use Wookie than to Download JDK with Java.

Background

WookieView allows you to automate web browsing – i.e. to search with Google Search, visit your Facebook News Feed, download files from sites. For people who are familiar with Selenium it is can a lightweight alternative to Selenium. It has very few external dependencies and uses WebView from JDK 8 to render pages.

Visually it is a browser window with almost none of the buttons:

The job of a developer is to create a script which would send commands to this window like mouse clicks and key presses. To find an element on a page CSS3 selectors are used.

Search Google Example

A simple script example to search with Google: 

load("http://www.google.com")

$("input[maxlength]")
  .value("wookie-view")
  .submit() // execution is blocked here during submit()

println("results: " + $("h3.r").asResultList)

An extended script example:

// first register the hook
wookie
    .waitForLocation(wookie.defaultArg("google search results")
    .matchByAddress((s) -> s.contains("q="))
    .whenLoaded(e -> {
        System.out.println("results: " + $("h3.r").asResultList());
    }));

// then open a page
load("http://www.google.com")

// submit google request and start the scenario
$("input[maxlength]")
    .value("wookie-view")
    .submit(n);

An extended API version is not just the same, but more verbose, it was meant to be a wrapping API for WebView, so that it replicates its funcitons more precisely. There might be few rare situations when you will require to use extened API over simple, but so far I was able to translate non-trivial cases like downloading a file into plain script versions. So a hypothetical complex scenario is when after a certain action over a browser state (i.e. entering a password and clicking ok), the resulting state is not determined by a page-ready event (i.e. a file download starts).

Star a GitHub Project

This script runs a google search, opens a GitHub page, logins into it and stars the project. Guess which project this is :)

The full script is available at GitHub.

load("http://www.google.com")

$("input[maxlength]")
  .value("wookie-view")
  .submit()

System.out.println("results: " + $("h3.r").asResultList)

// get the list of result links with CSS selector,
// find a link in it and follow it
$("h3.r a").asResultList().find(_.text().contains("chaschev")).get.followLink()

// it should be a GitHub page, click a sign in button
$("a.button.signin").followLink()

// fill login data at the login page
$("#login_field").value(login)
$("#password").value(password)
  .submit()

// find the star button after login
val starButton = $(".star-button:visible")

// check if it is already clicked and click if needed
if(starButton.text().contains("Star")) {
  starButton.mouseClick()
  println("Now the new star is born!")
} else {
  println("The stars all shine!")
}

Downloading JDK

This example shows how to:

• Download JDK from the Oracle Website
• Handle a file download
• Run custom JavaScript code

The full program text is available at GitHub.

Running custom JavaScript

This JavaScript is used to retrieve a list of JDK links on the page. It was created for the old version of Wookie which didn't have selectors on the Java side, with the new version it might be easier to ask for a list of links with selectors Java API. For demonstration and author's time management purposes the old version is used.

You can specify *.js files to add during Wookie construction:

val wookieView = WookieView.newBuilder
  .useFirebug(false)
  .useJQuery(true)
  .createWebView(!DownloadJDK.miniMode)
  .includeJsScript(io.Source.fromInputStream(getClass.getResourceAsStream("/wookie/downloadJDK.js")).mkString)
  .build

Then you just call a method declared in this file as you would call JS from Java. You can use a helper function for this:

val jQueryObj = FXUtils.execInFxAndAwait(() => {
   browser.getEngine.executeScript("find('" + DownloadJDK.version + "', true, 'linux');").asInstanceOf[JSObject]
})

Handle a file download

To add a file download, one needs to add a download hook. It can be added anytime before the actual download starts and will be removed when is matched. You can even add it in the beginning of the script, but make sure that some two hooks won't fire simultaneously.

// this creates a download hook which will fire by an address criteria
val downloadFuture = addDownloadHook(new LocationMatcher(loc =>
  loc.contains("download.oracle.com") && loc.contains("?")
))

jQueryObj.get.followLink()  // follow the download link

// download will start now and a file will be saved to a folder
// you can track it by using a future returned by addDownloadHook
// when download finishes, execution will continue to the next line

The full program listing for downloading JDK is available at GitHub. Though the code might seem huge and a little confusing, this new version is 3-4 times shorter than the one which uses plain WebView.

Summary

WookieView is written as an easy-going alternative to the WebView. It can also be seen as a lightweight Selenium alternative for which you will need very few external dependencies. With WookieView you can do: integrational testing of your Web UI, browsing a site, crawling the Web, downloading a file behind login protection, automating web operations, etc.

Right now it is a Scala+Maven project, more Scala than Java, there is an example in Java, but the Java API is raw. If interest in the project is shown, there will be work done to better interoperate with Java and I will also publish artifacts into Maven Central. 

If you have questions about how to use it or suggestions on how to improve it, feel free to write to me or submit bugs at the GitHub page.