Requestor development: a quick primer

This tutorial shows how to use the new Golem to run an app inside a custom Docker image in parallel on multiple providers.

Prerequisites

Platforms

While it's possible that you'll be successful running Golem and this tutorial on other platforms, we officially support the following:

  • OS X 10.14+

  • Ubuntu 18.04 or 20.04

  • Windows

Languages

If you are JS developer, please click to NodeJS tab

Python
NodeJS
Python

Python 3.6+

To verify your currently installed version of python, please run:

python3 --version

If you have an older version of python and you'd like to keep that version in your system, consider using pyenv. You can use pyenv-installer to facilitate the process.

On Windows, you may need to just use python instead of python3

NodeJS

Please bear in mind that our JS/TS API is an early alpha release and not nearly as refined as the Python one.

NodeJS 12.13.0+

To verify your currently installed version of node, please run:

node --version

If you have an older version of node and you'd like to keep that version in your system, consider using nvm. You can install it using the instructions from:

Yarn 1.22.3+

Verify that with:

yarn --version

If you don't have yarn or need to update, it, please go to: https://classic.yarnpkg.com/en/docs/install/ and choose the version appropriate for your operating system.

Git

You'll also need the git versioning system client so you can clone our repositories. Ensure you have it available with:

git --version

No crypto assets needed (for now!)

Alpha lives on the Rinkeby Testnet. You don't need any real ETH or GNT tokens to start this tutorial. You also don't need to do anything to get test tokens! These test assets are acquired by the daemon in one of the steps below.

Can we help you? Do you have feedback for Golem?

If you'd like to give us feedback, suggestions, have some errors to report or if you got stuck and need help while following our tutorials, please don't hesitate to reach out to us on our Golem Discord: https://chat.golem.network

Running the yagna daemon

Yagna is the main service of the new Golem that's responsible for keeping connections with all the other nodes in the network.

In order to follow our requestor agent tutorial, you'll first need to run the yagna daemon.

Easy installation

You can install it using our helper script like this:

curl -sSf https://join.golem.network/as-requestor | bash -

You might be asked to modify your PATH afterwards.

On Windows, only the manual installation is supported.

Manual installation

Alternatively, if you'd like to have more control over the installation process, or would like to choose where the binaries end up, you can do that manually.

First, download the requestor package - prefixed golem-requestor - appropriate for your platform from: https://github.com/golemfactory/yagna/releases/tag/v0.4.0

Unpack it and put the binaries contained within somewhere in your PATH (e.g. copy them to /usr/local/bin on unix-like systems) or add the directory you placed the binaries in to your PATH.

It's important for the yagna and gftp binaries to be available in your shell's PATH, otherwise, you'll encounter issues while continuing the tutorial.

Confirm the installed daemon's version

Once binaries are installed, confirm that you're running the latest Golem release:

yagna --version

It should output: yagna 0.4.0-a1a50fd9

Purge the stale working directories

If you had run a previous version of yagna in the past, you'll need to purge its working directories since our newest version is incompatible with the old database structure:

Ubuntu
mac OS X
Windows
Ubuntu
rm -rf $HOME/.local/share/yagna
mac OS X
rm -rf $HOME/Library/Application\ Support/GolemFactory.yagna
Windows
rmdir /s %APPDATA%\GolemFactory\yagna

Run the daemon

Now, you can run the daemon:

yagna service run

Important: After you launch the daemon, leave it running in the background while you proceed with the tutorial.

You can now proceed to Generate the app key.

Warning! Construction zone: errors ahead

You may notice errors while running the yagna daemon or the example script. Some of those errors will be silenced, prevented or handled more gracefully in the future production version. For now, you can usually ignore them, unless of course, they make the daemon or the example script crash - or - cause the task itself to fail before completion.

Some of the errors you may encounter are:

[2020-08-11T10:49:17Z ERROR ya_service_bus::remote_router] bind error: already registered: Service ID [... url ... ] already registered

Task exception was never retrieved [... several lines of stack trace ...] {"message":"GSB error: bad request: No service registered under given address"}

Generate the app key

With the daemon running, enter the daemon's directory using another shell and generate the yagna app key that will be used by your requestor agent to access yagna's REST API.

yagna app-key create requestor

This should produce a 32-character-long hexadecimal app key that you need to note down as it will be needed to run the requestor agent.

In case you lose your app key, you can retrieve it with:

yagna app-key list

the value in the key column is the key you need.

Enable the daemon as a requestor

You need the following command to enable the daemon as a requestor.

What it also does under the hood, it also checks for funds on your requestor node and if needed, contacts the faucet which issues some ETH and nGNT tokens to the node.

It needs to be run each time the daemon is started or restarted.

yagna payment init -r

Once you issue the command, allow some time until it completes its job.

Due to the fact that our example uses the Rinkeby Ethereum testnet, the transactions that add ETH and nGNT to the requestor node's address need to be mined and confirmed there. It may take several minutes until that's completed.

You can verify whether you already have the funds with:

yagna payment status

If, after a few minutes, you can't see the assets, re-run the payment init command above and check again after a few more minutes.

Running the requestor and your first task on the New Golem Network

Now you have the yagna daemon running, you may proceed with running a task as a requestor.

Python
NodeJS
Python

Get the environment set up

Ensure you're running python >= 3.6 and you have the venv module installed (it's normally included in the python distribution).

Prepare a virtual environment for the tutorial script:

python3 -m venv ~/.envs/yagna-python-tutorial
source ~/.envs/yagna-python-tutorial/bin/activate

On Windows, you need to replace the above with:

python -m venv %HOMEDRIVE%%HOMEPATH%\.envs\yagna-python-tutorial
%HOMEDRIVE%%HOMEPATH%\.envs\yagna-python-tutorial\Scripts\activate.bat

Install the dependencies:

pip3 install -U pip
pip3 install certifi yapapi

Get the requestor agent's code

Check out or download the yapapi repository:

git clone https://github.com/golemfactory/yapapi.git

and make sure you're working on the version corresponding with the latest release:

cd yapapi
git checkout b0.3

Set the yagna app key

In order for the requestor agent to connect with the yagna daemon, you need to provide it with the previously-generated app key. You do that by setting the appropriate environment variable to a value acquired in the "Generate the app key" step above:

export YAGNA_APPKEY=insert-your-32-char-app-key-here

On Windows, please replace the above with:

set YAGNA_APPKEY=your-32-char-app-key

Run the example

The example we're showcasing here resides in the examples/blender directory within yapapi's codebase so, ensure that you're in the checked-out repository's directory and run:

cd examples/blender
python3 blender.py --subnet-tag devnet-alpha.2

Once you launch the example, you should see some messages reflecting the progress of your task's execution - agreement confirmations, task dispatches and finally task completions.

The example in question generates six discrete jobs for providers to execute so after those six activities are completed and results returned, the whole task is finished.

If everything goes right, after what could be anything from half-a-minute to a few minutes, you'll hopefully see the message announcing the successful completion of your assignment including a short summary of what had happened during the execution, which providers took part in the execution and the accumulated GNT cost of the whole task, e.g.:

Computation finished in 77.5s Negotiated 1 agreements with 1 providers Provider 'odra' computed 6 tasks Total cost: 0.218290307253

NodeJS

Get the requestor agent's code

Check out or download the yajsapi repository:

git clone https://github.com/golemfactory/yajsapi.git
cd yajsapi
git checkout b0.1

Set the yagna app key

In order for the requestor agent to connect with the yagna daemon, you need to provide it with the previously-generated app key. You do that by setting the appropriate environment variable to a value acquired in the "Generate the app key" step above:

export YAGNA_APPKEY=insert-your-32-char-app-key-here

On Windows, please replace the above with:

set YAGNA_APPKEY=your-32-char-app-key

Run the example task

The example we're showcasing here resides in the examples/blender directory within yajsapi's codebase so, ensure that you're in the checked-out repository's directory and run:

cd examples
yarn
yarn js:blender

If everything works as expected, you should see some messages that confirm agreements being struck between your requestor node and the providers in our testnet and then ones that announce work dispatched to providers with lines starting with new batch !!! and subsequently confirmations of task completions.

To some more detailed messages, you can run the example with yarn js:blender -d.

The example in question generates six discrete jobs for providers to execute so after those six activities are completed and results returned, the whole task is finished.

progress= {'done': True}

Yay! With this, you have completed your first job as a requestor in the new Golem network!

Output

You can verify that the task is indeed done by examining the generated output files which are PNG images with the selected frames of the rendered animation that should appear in the directory from which you ran the example script (examples/blender within the cloned repository's path if you followed the tutorial precisely) .

Here is an example rendered frame, provided here for reference:

Payments

Finally, you can verify that the providers have been paid for the work they contributed to get that output to you. First, acquire your Ethereum address - you can do that by running:

yagna app-key list

again but this time it's the value in the id column that you're interested in. This is your the Ethereum address of your yagna node on the Rinkeby testnet. Once you have that address, head to https://rinkeby.etherscan.io/ and put the value in the address field there. Afterwards, switch to the "Erc20 Token Txns" tab and you should be able to see your outgoing payments there.

Next steps

So, you have successfully completed your first task as a requestor on the new Golem network. Are you curious to learn more? Wanna do more stuff?

If you'd like to understand, extend and play around with our example, please consult:

On the other hand, if you'd like to deploy your own dockerized apps to our alpha testnet, have a look at: