JS Task API Examples: accessing the internet

Introduction

In this article, we will present methods that let you access the Internet.

Outbound feature

For the requestor to be able to use the outbound feature, (initiate a connection to target_url), the following minimal conditions must be met:

The requestor must request the outbound feature in the demand and include a Computation Manifest there. The manifest must declare the target_url.
The provider offers the service at least for the target_url. (So either outbound for unrestricted URLs or the target_url is included in the whitelist).

You can find more information about the feature and the manifest in the following articles: Accessing the Internet and Payload Manifest.

The requestor is responsible for:

Manifest creation
Defining demand for outbound access

Both examples are accessing urls included in the Whitelist. For the current Whitelist see https://github.com/golemfactory/ya-installer-resources/tree/main/whitelist.

Prerequisites

Yagna service is installed and running with try_golem app-key configured.

How to run examples

Create a project folder, initialize a Node.js project, and install the @golem-sdk/task-executor library.

mkdir golem-example
cd golem-example
npm init
npm i @golem-sdk/task-executor

Next, install Golem SDK CLI - a companion tool that will facilitate manifest creation.

npm install -g @golem-sdk/cli

To run the examples provided below, copy the code into the index.mjs file in the project folder and run:

node index.mjs

Basic outbound access

In this example we will download a file from ipfs.io. This is one of the domain included in the whitelist.

Manifest creation

To create a new manifest run:

golem-sdk manifest create golem/examples-outbound:latest

This will create a basic manifest.json file. You will use it to inform the provider what GVMI image we will be using. The manifest contains also your application version, application name, and description, all read from your package.json file (you can edit this information if you want).

Adding outbound configuration

The next step is to configure our manifest, so you can access a public URL. The CLI also has a handy command that will take care of that for you:

golem-sdk manifest net add-outbound https://ipfs.io

This has added 'https://ipfs.io' as the URL you want to access from the provider node. The command can be run multiple times to add more URLs or you can pass them all at once.

Your manifest is ready and stored in the manifest.json file.

Defining demand for outbound access

The example below demonstrates how to define the demand that will get access to the Internet.

Here's the manifest

{
  "version": "0.1.0",
  "createdAt": "2023-10-19T11:23:08.156+02:00",
  "expiresAt": "2024-01-17T11:23:08.156+01:00",
  "metadata": {
    "name": "outbound-docs",
    "description": "",
    "version": "1.0.0"
  },
  "payload": [
    {
      "platform": {
        "os": "linux",
        "arch": "x86_64"
      },
      "hash": "sha3:dad8f776b0eb9f37ea0d63de42757034dd085fe30cc4537c2e119d80",
      "urls": [
        "http://registry.golem.network/download/f37c8ba2b534ca631060fb8db4ac218d3199faf656aa2c92f402c2b700797c21"
      ]
    }
  ],
  "compManifest": {
    "version": "0.1.0",
    "net": {
      "inet": {
        "out": {
          "urls": ["https://github.com", "https://ipfs.io"],
          "protocols": ["https"]
        }
      }
    }
  }
}

And the requestor code:

import { TaskExecutor, pinoPrettyLogger } from "@golem-sdk/task-executor";
import { readFile } from "fs/promises";

// The example is using url from domain that is included in the outbound Whitelist.
// See https://github.com/golemfactory/ya-installer-resources/tree/main/whitelist for the current default whitelist.

const url = "https://ipfs.io/ipfs/bafybeihkoviema7g3gxyt6la7vd5ho32ictqbilu3wnlo3rs7ewhnp7lly";

(async function main() {
  // Load the manifest.
  const manifest = await readFile(`./manifest.json`);

  // Create and configure a TaskExecutor instance.
  const executor = await TaskExecutor.create({
    capabilities: ["inet", "manifest-support"],
    yagnaOptions: { apiKey: "try_golem" },
    logger: pinoPrettyLogger(),
    manifest: manifest.toString("base64"),
  });

  try {
    await executor.run(async (ctx) => {
      const result = await ctx.run(`curl ${url} -o /golem/work/example.jpg`);

      console.log((await ctx.run("ls -l")).stdout);
      if (result.result === "Ok") {
        console.log("File downloaded!");
      } else {
        console.error("Failed to download the file!", result.stderr);
      }
    });
  } catch (err) {
    console.error("The task failed due to", err);
  } finally {
    await executor.shutdown();
  }
})();

Note the most important part:

// Load the manifest file.
const manifest = await readFile(`./manifest.json`)

// Create and configure a TaskExecutor instance.
const executor = await TaskExecutor.create({
  capabilities: ['inet', 'manifest-support'],
  yagnaOptions: { apiKey: 'try_golem' },
  manifest: manifest.toString('base64'),
})

First, it is specifying additional requirements to the demand:

'inet' - indicates the script requires outbound service
'manifest-support' - indicates, requestor uses a manifest to specify a demand.

Instead of providing an image tag or hash, it uses a manifest file that describes what will be run on providers.

Please note the loaded manifest is encoded to base64.

yagnaOptions: { apiKey: 'try_golem' } - defined the api key, to get access to the Yagna service. This particular key is available if you start the yagna according to the procedure provided in the installation example, you can also configure your own unique keys. See here for instructions.

Then you can use the applications that connects to the target url specified in the manifest in the standard way:

const result = await ctx.run(`curl ${url} -o /golem/work/example.jpg`)

Using outbound to install node.js packages

Note: This example shows how to use the outbound whitelist to install a npm package on a provider. It is recommended to install additional packages in a directory that is defined as VOLUME in the image definition, to avoid filesystem capacity limits. If you have a large number of packages you should rather install them during the image build phase - you will avoid installing them on each provider separately and get your providers ready in a shorter time.

Here's the manifest:

{
  "version": "0.1.0",
  "createdAt": "2023-11-28T12:23:33.435+01:00",
  "expiresAt": "2024-02-26T12:23:33.435+01:00",
  "metadata": {
    "name": "npm_install",
    "description": "Example project",
    "version": "1.0.0"
  },
  "payload": [
    {
      "platform": {
        "os": "linux",
        "arch": "x86_64"
      },
      "hash": "sha3:3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd",
      "urls": [
        "http://registry.golem.network/v1/image/download?hash=3d6c48bb4c192708168d53cee4f36876b263b7745c3a3c239c6749cd"
      ]
    }
  ],
  "compManifest": {
    "version": "0.1.0",
    "net": {
      "inet": {
        "out": {
          "urls": ["https://registry.npmjs.org"],
          "protocols": ["https"]
        }
      }
    }
  }
}

And the requestor code:

import { TaskExecutor, pinoPrettyLogger } from "@golem-sdk/task-executor";
import { readFile } from "fs/promises";

const manifest = await readFile(`./manifest_npm_install.json`);

(async function main() {
  const executor = await TaskExecutor.create({
    // What do you want to run
    capabilities: ["inet", "manifest-support"],
    manifest: manifest.toString("base64"),

    yagnaOptions: { apiKey: "try_golem" },
    budget: 0.5,
    logger: pinoPrettyLogger(),

    expires: 1000 * 60 * 30, //h

    // Control the execution of tasks
    maxTaskRetries: 0,

    taskTimeout: 120 * 60 * 1000,
  });

  try {
    await executor.run(async (ctx) => {
      console.log("working on provider: ", ctx.provider.id);

      console.log((await ctx.run("npm install moment")).stdout);
      console.log((await ctx.run(`cat ./package.json`)).stdout);

      return 1;
    });

    console.log("task completed");
  } catch (err) {
    console.error("Running the task on Golem failed due to", err);
  } finally {
    await executor.shutdown();
  }
})();