Custom Javascript

Custom Javascript can be injected in an X-Forward-X-Evaluate custom header of an Extract API call, or inserted into a Custom API Rule via the API Dashboard, but it has strict requirements in regards to syntax.

• an X-eval script starts with an anonymous function declaration which will be called by Diffbot

  JavaScript

  function () {}
  

• an X-eval script needs to start with start() and end with end(). In particular, if end is omitted, the script never ends and the request will time out waiting for end() to be called.

  JavaScript

  function () {start(); end();}
  

• each line in an X-eval script needs to end in a semi-colon to ensure accurate Javascript parsing.

For performance reasons, we recommend minifying your Javascript (turned into a single-line) using a minifier like Minifier.org, and then that minified code can be inserted after the start() function in the function definition above. The entire string may then be injected into a custom API ruleset via your account Dashboard, or injected into an API call made via GET or POST (via the X-Forward-X-Evaluate header).

For ease of use, jQuery is supported with this feature in addition to native Javascript.

X-Evaluate Example

The web page Example.com has a paragraph of text explaining its purpose. Let's add a "Today is X" message to the end of that paragraph, with X being the current day of the week.

These are the steps we should follow:

1. Write a script which successfully does this in the browser console.

   JavaScript

   const now = new Date();
   const days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];
   const day = days[ now.getDay() ];
   document.querySelector("p").innerText += "Today is " + day;
   

2. Minify this script using Minifier.org and wrap it in function and the start and end calls.

   JavaScript

   function() {start(); const now=new Date,days=["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"],day=days[now.getDay()];document.querySelector("p").innerText+="Today is "+day; end(); }
   

3. Pass the script along as the X-Forward-X-Evaluate custom header. Here is an example using curl:

   Shell

   curl --location 'http://api.diffbot.com/v3/article?token=TOKEN&url=http%3A%2F%2Fexample.com' \
   --header 'X-Forward-X-Evaluate: function() {start(); const now=new Date,days=["Sunday","Monday","Tuesday","Wednesday","Thursday","Friday","Saturday"],day=days[now.getDay()];document.querySelector("p").innerText+="Today is "+day; end(); }'
   

Applying Custom Javascript via a Custom API

Using the Custom API functionality in the Dashboard at we can apply our X-Evaluate Custom Javascript function to a ruleset, such that it would be applied whenever a URL matching a particular regex (covering the whole domain by default) was called with a particular Extract API. This can be accomplished as follows:

1. Browse to https://app.diffbot.com/custom/create/
2. Enter the URL http://example.com and select Article API
3. Click the "Other Settings" tab at the top of the page and scroll down to the "X-Evaluate" field.
4. Enter the code from the X-Evaluate example above:

const now = new Date();  
const days = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'];  
const day = days[ now.getDay() ];  
document.querySelector("p").innerText += "Today is " + day;

5. Click Minify to minify the code. Note that this can also be used as an alternative to using Minifier.org in the example above.
6. Wrap the code in your function/start/end calls as in the previous example.
   
7. Save the form.

Now whenever a page is called on the example.com domain with the Article API using your token, the custom Javascript function will be run prior to extraction.

📘

X-Evaluate will only be executed if called from the API the rule resides in

If you have an X-Evaluate script in your Article API rule and make a request with the Analyze API that identifies the page as an article, the X-Evaluate will not be executed.

Saving data directly to a JSON output field

Javascript data types and data structures can be saved directly to a named field in the JSON result, using the save function as follows:

save(fieldName, fieldValue)

The example below grabs the cookie from the browser and outputs it to a field called theCookie:

function() {
    start();
    save("theCookie", document.cookie);
    end();
}

Minified, this would look like:

function(){start(),save("theCookie",document.cookie),end()}