Service Example 2: Erigolem – Managed Erigon

Ethereum client as a service, implemented using a self-contained runtime

The purpose of this example is to demonstrate building a Golem service using a dedicated, self-contained runtime. It features an Ethereum network client, Erigon (f.k.a. Turbo-geth) started on provider nodes on user's demand. Interaction with the service, once it's running and its location and credentials have been passed to the requestor, is not facilitated by Golem though.

This example illustrates the following Golem features & aspects:

All code for this example can be found in this repository:

Architecture overview

The application consists of three major components:

  1. User interface

    A browser-based interface written in TypeScript using React. It allows the end user to interact with the application, i.e. start an Erigon node after authenticating with their Metamask wallet, and later to manage their node(s).

  2. Requestor server

    A Python application developed using Quart and yapapi-service-manager. It acts both as an HTTP server (handling requests from the user interface), and as a requestor agent (submitting tasks to the Golem network).

  3. Erigon runtime

    A dedicated, self-contained runtime created with ya-runtime-sdk. It is a Rust binary wrapping Erigon service itself so that it can be orchestrated by the yagna daemon. The runtime also controls the access to the service by managing Nginx configuration.

A typical interaction flow proceeds in the following manner:

  1. User opens the web interface in their browser.

  2. User authenticates with their metamask wallet.

  3. User clicks to start an Erigon node.

  4. Request to start an Erigon node is sent via the REST API to the requestor server.

  5. Requestor starts a service activity (these steps are handled by yagna core and yapapi internally):

    1. Requestor server makes a number of API calls to the yagna daemon running on the requestor's machine in order to find an available Erigon provider and sign an agreement.

    2. Once the agreement is signed, requestor server sends a command to the provider to start an Erigon node.

    3. Yagna daemon running on the provider's machine spawns an Erigon runtime process (a service wrapper/supervisor process).

  6. Erigon runtime starts the actual Erigon service (backend & rpcdaemon processes).

  7. Erigon runtime generates a new username and password and puts them in the NGINX configuration.

  8. NGINX auto-reloads on configuration change, from now on granting access to the Erigon service only for user(s) authenticated with the newly-created credentials. (NGINX is continuously running on provider's machine as a daemon, so it doesn't need to be started.)

  9. The credentials are passed backed to the requestor server along with provider's node's public IP address and the port on which NGINX is listening.

  10. Requestor server passes all the received information to the web interface where it can be viewed by the user.

  11. User interacts with the Erigon service (via NGINX proxy).

  12. Once the user finishes using their Erigon instance, they click to stop it.

  13. Web interface sends a request to stop the instance via the REST API to the requestor server.

  14. Requestor server sends a stop command to the provider.

  15. Erigon runtime stops the Erigon service and exits.

  16. Requestor server orchestrates a payment for the service in the background without further user interaction.

The design presented above excludes making any payments for the service by its end user. This omission was made intentionally to make te service implementation simpler. It's not a problem as long as the service is used for demonstrative purpose only.

Runtime implementation

Here we demonstrate how self-contained Exe-Unit runtimes can be implemented, tested and finally plugged into the Provider's daemon.

Please be aware that a custom runtime doesn't provide the level of isolation comparable with a virtual machine. It's just an executable running with the same privileges as the yagna daemon.

Erigon runtime

First, let's discuss what the purpose of the Erigon runtime is to gain some high-level overview. The runtime:

  • Accepts an Ethereum network's name (e.g. 'mainnet' or 'rinkeby') from the requestor agent,

  • Starts Erigon binaries to sync with the chosen network and serve Erigon RPC endpoint,

  • Generates credentials for basic authentication which is implemented with nginx,

  • Returns the credentials to the requestor agent so the user of the service can connect to the Erigon RPC.

Please refer to the "Provider's node" in the above Architecture overview diagram.


Before diving into this section you might want to review simpler example runtime available in the ya-runtime-sdk repository.

Erigon runtime declares two structures

  • ErigonConf - containing runtime configuration with default values,

  • ErigonRuntime - derives from RuntimeDef implements the required methods from ya-runtime-sdk::Runtime trait

At least these methods from the Runtime trait have to be implemented but Runtime SDK provides default implementation to several others.


deploy - It's run only once per activity, does not accept parameters (but there are plans to add them in the future release), return value is not passed to the requestor agent. Requestor agent is not required to call deploy directly, in this case deploy and start will be called automatically by the SDK before the first command sent to the runtime.

In the Erigon runtime this method asserts that parent directory for Erigon data provided via the configuration exists and then creates directories for each Ethereum network supported by the service.

Let's go through it step by step

Service configuration struct is defined as follows, see the data_dir property below.

#[derive(Deserialize, Serialize)]
pub struct ErigonConf {
public_addr: String,
data_dir: String,
passwd_tool_path: String,
// ...

We were able to declare most properties with the type of String not Option<String> because we also implemented the Default trait for ErigonConf struct.

Now in the deploy we access data_dir property of EriginConf struct from Context parameter conf property.

fn deploy<'a>(&mut self, ctx: &mut Context<Self>) -> OutputResponse<'a> {
let data_dir = PathBuf::from(&ctx.conf.data_dir);
// check if parent data directory exists
if !(data_dir.exists() && data_dir.is_dir()) {
panic!("Configuration 'data_dir' directory has to exists...");

We declared supported networks of the Ethereum in the Network enum

custom_derive! {
#[derive(Debug, PartialEq, EnumDisplay, EnumFromStr, IterVariantNames(NetworkVariantNames))]
pub enum Network {

So the next part is to iterate through Network's variant names and create a subdirectories for each of the network.

// create all supported chains data subdirs
for chain_subdir in Network::iter_variant_names()
.map(|chain| chain_subdir(data_dir.as_path(), chain)) {
if !&chain_subdir.is_dir() {
.unwrap_or_else(|_| panic!("Unable to create subdir {:?}", &chain_subdir));

Thanks to ya-runtime-std all is left is to communicate success, by returning Ok(None) to the SDK.


start - called after deploy usually once per activity, accepts parameters. Analogically with deploy if not run explicitly via the SDK deploy and start without parameters are run in order before any command is send to the runtime.

In the Erigon runtime we use start method to receive the network parameter from the requestor, start Erigon's binaries and generate new credentials for nginx basic auth. As above, we'll review the code more deeply.

First, we receive the input from the caller from context ctx.cli.command.args() provided by SDK's CommandCLI trait Command enum. Arguments to start are passed as a &Vec<String>, we're interested only of the first of them. get_chain_from_config is retrieving the network value from the json and converts it to the Network variant, to assert it's within supported networks. We're also setting its lowercased String value in the erigon_chain property for the later use.

fn start<'a>(&mut self, ctx: &mut Context<Self>) -> OutputResponse<'a> {
let chain = get_chain_from_config(ctx.cli.command.args().into_iter().next());
let chain = chain.to_string().to_lowercase();
self.erigon_chain = Some(chain.clone());

Next, we're generating new password for the nginx's basic auth. We're generating random string, set it with the passwd_tool and store it in the erigon_password service's property.

// Generate user & password entry with passwd tool
let rng = thread_rng();
let password: String = rng.sample_iter(Alphanumeric)
// ...
let _ = generate_password_file(
.expect("Unable to create passwd file");
self.erigon_password = Some(password);

What's left is to spawn erigon and rpcdaemon processes and store handles to them so we can kill them when the service will be stopped.

let mut erigon_pid = spawn_process(
&mut Command::new(&path.join(ERIGON_BIN))
&[""; 0],
.expect("Erigon: Failed to spawn");
let rpcd_pid = spawn_process(...);
self.erigon_pid = Some(erigon_pid);
self.rpcdaemon_pid = Some(rpcd_pid);

The start's return value is serving only troubleshooting purposes as the Erigon agent ignores it but it can be reviewed in the debugger session.

Note that the difference between deploy and start regarding the Erigon runtime might not be obvious. It's easier to think of them in terms of VM runtimes where deploy is needed for runtime preparation, e.g. download the image, after start runtime should be ready to run the commands.

Run command

run_command - called every time the corresponding agent SDK run is called on the WorkContext object. Result can be obtained from the WorkContext's results array after sequence of commands is executed by commit.

In the Erigon runtime this method is used to provide Erigon service's credentials to the agent. Here we're just forming the JSON string response and passing it as a command output, see the call of run_ctx.stdout(...) below.

let erigon_status_data = serialize::json::json!({
"url": &ctx.conf.public_addr,
// ...
ctx.command(|mut run_ctx| async move {
let stdout = format!("{}", erigon_status_data);


stop - can be called explicitly by the requestor agent or automatically due to agreement termination.

In the Erigon runtime stop method is used to kill Erigon processes started in start.

let erigon_kill_ok = match &mut self.erigon_pid {
Some(erigon_pid) => erigon_pid.kill().is_ok(),
None => false,
// kill rpc_daemon process bellow


Above methods corresponds to CLI::Command enum. One can extend the default CLI by

  • Deriving StructOpt on the CLI struct,

  • Decorating the runtime struct with #[cli(<struct_name>)].

Custom CLI arguments will be passed to each command by runtime::Context.

Default RuntimeMode is Server. In this mode runtime is deployed and then communicates with the ExeUnit via the Runtime API. Another option is RuntimeMode::Command where each command is a separate invocation of the runtime binary.

Runtime configuration

During runtime startup the configuration file located in $HOME/.local/share/ya-runtime-erigon/ya-runtime-erigon.json is parsed and is initialising the instance of ErigonConf structure. If the file does not exists it will be created and initialized with default values during the first start of the runtime.

Example configuration file

"public_addr": "",
"data_dir": "/data/erigon",
"passwd_tool_path": "htpasswd",
"passwd_file_path": "/etc/nginx/erigon_htpasswd",
"password_default_length": 15,
"erigon_http_addr": "",
"erigon_http_port": "8555"

Testing the runtime

Once needed SDK methods are implemented we can give our runtime a try in the debugger. For example our Erigon runtime can be executed in the debugger shell running the following command.

ya-ya-runtime-dbg \
--runtime ./target/debug/ya-runtime-erigon \
--workdir ./target/debug \
--start-arg '{"network": "goerli"}'

Please mind it requires Erigon binaries present in the same directory (all needed binaries for Ubuntu Linux can be downloaded from the binary release). Also htpasswd from the apache-tools is needed to be present in the PATH but for the local testing you can mock it with echo changing configuration file property "passwd_tool_path": "echo",

Pluging the runtime into golemsp

In the $HOME/.local/lib/yagna/plugins/ directory create

  • file ya-runtime-erigon.json where you describe the plugin

    "name": "erigon",
    "version": "0.1.0",
    "supervisor-path": "exe-unit",
    "runtime-path": "ya-runtime-erigon/ya-runtime-erigon",
    "description": "Service wrapper for Erigon (formelly Turbo-Geth)",
    "extra-args": ["--runtime-managed-image"]
  • directory ya-runtime-erigon (compare runtime-path in above file) where ya-runtime-erigon binary along with Erigon binaries are placed.

New runtime needs also to be enabled in $HOME/.local/share/ya-provider/presets.json. Preset object can be copied from other presets. Please note that exeunit-name has to match to the name property of the plugin above

"active": [
"presets": [
"name": "erigon",
"exeunit-name": "erigon",
"pricing-model": "linear",
"usage-coeffs": { ... }

Requestor agent

This part of the tutorial directly corresponds to the two previous requestor tutorials (services hello world example and simple service). We have the same clear separation between service specification and service provisioning, but there are important differences:

  • We use Erigon runtime instead of a VM-based runtime (so we don't have any Dockerfile or image_hash).

  • We don't implement async def run - the service is only started/stopped, requestor is idle when the service it is running.

  • We use yapapi-service-manager instead of pure yapapi.

  • We integrate the requestor code with Quart, a web framework.

Service specification

As in previous tutorials, service specification is a subclass of yapapi.Service. Full code is available in the yagna-erigon-repo, here we'll discuss the important parts.


class Erigon(Service):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.url = None
self.auth = None = None

We add three attributes that are specific to our service logic:

  • url - where Erigolem RPC (running on the provider) will be accepting requests

  • auth - credentials that must be provided with each request

  • network - Ethereum network name

They are initialized to None and later - when the service starts - will be updated with the data acquired from the runtime.

We have no control of the Erigon service class initialization, so the

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)

part is required.


The payload is defined in ErigonPayload class:

class ErigonPayload(Payload):
runtime: str = constraint(inf.INF_RUNTIME_NAME, "=", ERIGON_RUNTIME_NAME)
min_mem_gib: float = constraint(inf.INF_MEM, ">=", 0.5)

and used in get_payload in the service:

async def get_payload(cls):
return ErigonPayload()

Few important things to note here:

  • We don't have any image_hash (contrary to the previouse examples) because we don't use a VM-based runtime.

  • We declare the runtime name, erigon - this must match the offered exeunit-name.

  • min_mem_gib - minimum amount of RAM the provider has to offer. This is pretty useless in the Erigon case (contrary to the e.g. VM-based runtimes), but is required because of a known yapapi bug.


async def start(self):
# startup - set start args (network parameter)
start_args = await self._get_start_args()
if start_args:
erigon_init_args = start_args[0]
erigon_init_args_str = json.dumps(erigon_init_args)
# Set url & auth'STATUS')
processing_future = yield self._ctx.commit()
result = self._parse_status_result(processing_future.result())
self.url, self.auth, = result['url'], result['auth'], result['network']

Let's split that to separate parts.



This is the first thing that should always be done with self._ctx.

Determine start args

start_args = await self._get_start_args()

start_args - in the erigon case - are a single-item tuple:


These start_args are defined by the final user (the one ordering Erigon service, e.g. via the web interface) and passed directly to the runtime.

Current yapapi has no pretty way of passing arguments to the Service, so this is implemented as an ugly-but-harmless hack:

async def _get_start_args(self):
while True:
return self._cluster.instance_start_args
except AttributeError:
await asyncio.sleep(0.1)

This will be addressed in the near future.


if start_args:
erigon_init_args = start_args[0]
erigon_init_args_str = json.dumps(erigon_init_args)

start_args is expected to be a tuple, but there are no more assumptions - they are just passed here from the code that starts the service. The Erigon runtime expects at most one argument and it is expected to be a json, so we send the serialized first argument (or start without any arguments if start_args are empty). This could be also a good place to perform a requestor-side validation, we validate start_arg only in the runtime.

Perform a STATUS command'STATUS')

Run a command 'STATUS'. Actually, we could send anything other than STATUS - our runtime ignores the command and does always the same thing - returns running erigon url, credentials and Ethereum's network.

Compare this to VM-based commands, like'/bin/date')'/bin/sh', '-c', 'cat some_file.txt')


In the VM environment, available commands are defined in the image and they usually default to commands available in the base linux image + optional additional commands, like /golem/run/ in the simple service. When using a custom runtime, we are free to implement any commands we want - but also we have no built-in commands, not even ls.

Fetch the results

processing_future = yield self._ctx.commit()
result = self._parse_status_result(processing_future.result())
self.url, self.auth, = result['url'], result['auth'], result['network']
def _parse_status_result(self, raw_data: 'List[CommandExecuted]'):
command_executed = raw_data[-1]
erigon_data = command_executed.stdout
erigon_data = json.loads(erigon_data)
return erigon_data

STATUS command executed in the runtime returns JSON-serialized data. On the requestor side it is available in stdout of the appropriate command_executed.

processing_future.results() is a list of CommandExecuted objects - one per each command. In our case, there are three : deploy, start, and run(STATUS), and we are interested only in the output of the last one.

Run & shutdown

We don't need to implement those functions because the default yapapi.Service implementation is exactly what we want:

  • Default run() waits forever.

  • Default shutdown() terminates the agreement. We don't have to perform any additional cleanup - it is already implemented in the runtime (e.g. erigon process is stopped).

Running the HTTP Erigon server

We implement an HTTP server that responds to 3 requests:

  • createInstance - start a new erigon,

  • stopInstance/<id> - stop this erigon,

  • getInstances - list all current & past erigons.

There is also some basic authentication - the only user who can see/stop an erigon is the one who created it. We use the Quart framework.

Code that uses yapapi must be called in an asynchronous context. Quart is an async equivalent of the much more popular Flask.

Full server code is here, only Golem-related part will be discussed in this tutorial.

For the documentation on yapapi-service-manager check README.



app.yapapi_executor_config = {
'budget': 10,
'subnet_tag': os.environ.get('SUBNET_TAG', 'erigon'),

Here we define the yapapi.Golem init_args. app.yapapi_executor_config is passed directly to the Golem object.

Start/stop the requestor

from yapapi_service_manager import ServiceManager
async def start_service_manager():
app.service_manager = ServiceManager(app.yapapi_executor_config)
async def close_service_manager():
await app.service_manager.close()

We initialize the ServiceManager during the server startup and close it when the server exits. This is (roughly) equivalent to entering/exiting async with Golem.

There is no state preserved outside of the process memory, so when using a generic WSGI server (like gunicorn) you shouldn't

  • Start more than one worker,

  • Use worker-recycling tools (like max_requests in gunicorn).

Create a new Erigon

from .erigon_service import Erigon
from .erigon_service_wrapper import ErigonServiceWrapper
@app.route('/createInstance', methods=['POST'])
async def create_instance():
request_data, user_id, init_params = [some_non_golem_stuff]
erigon = app.service_manager.create_service(Erigon, (init_params,), ErigonServiceWrapper) = request_data.get('name', f'erigon_{}')
app.user_erigons[user_id][] = erigon
return erigon.api_repr(), 201

First, we extract request_data, user_id and init_params from the request, this is done in a pretty standard Quartish way.

Then comes the most important line:

erigon = app.service_manager.create_service(Erigon, [init_params], ErigonServiceWrapper)

Erigon is the yapapi.Service-based class we implemented in the previous section.

init_params is a dictionary {'network': <ETHEREUM-NETWORK-NAME>} that is defined in the request.

ErigonServiceWrapper is a class extending yapapi-service-manager.ServiceWrapper that can be found here. It is neither very important nor interesting: we just need a place to store and access some additional erigon-specific information, like created_at timestamp or name. This could be implemented in many different ways, but this is the most convenient - the returned erigon object (an instance of ErigonServiceWrapper) encapsulates all of the logic and has exactly the interface we need: # user-defined name
erigon.api_repr() # representation of the erigon, sent to the frontend
erigon.stop() # stops the service (service starts when the object is created)

In the next three lines: = request_data.get('name', f'erigon_{}')
app.user_erigons[user_id][] = erigon
return erigon.api_repr(), 201

We set the Erigon name, save the information about newly created Erigon, and send its representation as a response.

Stop the erigon

@app.route('/stopInstance/<erigon_id>', methods=['POST'])
async def stop_instance(erigon_id):
user_id = get_user_id()
erigon = app.user_erigons[user_id][erigon_id]
except KeyError:
return 'Invalid erigon_id', 404
return erigon.api_repr(), 200

Nothing really interesting here, we just:

  • Extract the user_id from the request

  • Check if this is the user who created this Erigon (compare the app.user_erigons[user_id][] = erigon line in the previous section),

  • Stop the Erigon - this initializes the stopping process, it is not stopped immediately (because stopping needs some action on the provider side),

  • Return Erigon representation as a response.

Get all user's Erigons

@app.route('/getInstances', methods=['GET'])
async def get_instances():
user_id = get_user_id()
erigons = app.user_erigons[user_id].values()
data = [erigon.api_repr() for erigon in erigons]
return json.dumps(data), 200

We select all Erigons created by the user and return their representation.

Running the service with a simple script

There is also a simple script that just starts the erigon services. Service is running forever, status is printed every second, ctrl+C leads to a graceful shutdown. This is just a development tool, similar to yapapi-service-manager examples.

Becoming an Erigolem provider

Here we describe an installation script which should make the whole process of becoming an Erigolem provider fun and smooth. The script is prepared for Ubuntu 18.04 or later and should work on other Linux distributions after some modifications.

This is an experimental setup. It's not intended to be run on personal computers because:

  • It creates a separate user which will run the provider daemon,

  • It installs the NGINX web server and certbot,

  • Optionally, it generates a Let's Encrypt certificate and sets certbot to remind you about certificate renewals,

  • It disables all runtimes for yagna other than ya-runtime-erigon.


  • Ubuntu 18.04 or later

  • Root privileges

  • Public IP address or a DNS name

  • Ability to open a port for the Erigon service

By default nginx's reverse-proxy is listening on the standard Erigon port 8545. You need to open this port for incoming traffic. If you wish your Erigolem service to be available on a different port number (e.g. because 8545 cannot be opened), you need to edit the installation script.

Example patch changing 8545 to 9999

diff --git a/ b/
index 2bb4de9..41ad5e2 100644
--- a/
+++ b/
@@ -161,8 +161,8 @@ http {
server {
- listen 8545${ERIGON_USE_SSL:+ ssl http2};
- listen [::]:8545${ERIGON_USE_SSL:+ ssl http2};
+ listen 9999${ERIGON_USE_SSL:+ ssl http2};
+ listen [::]:9999${ERIGON_USE_SSL:+ ssl http2};
server_name $ERIGON_HOSTNAME;
@@ -240,7 +240,7 @@ mkdir -p "${ERIGON_USER_HOME}/.local/share/ya-runtime-erigon"
cat >"${ERIGON_USER_HOME}/.local/share/ya-runtime-erigon/ya-runtime-erigon.json" <<EOF
- "public_addr": "https://${ERIGON_HOSTNAME}:8545",
+ "public_addr": "https://${ERIGON_HOSTNAME}:9999",
"data_dir": "${ERIGON_DATADIR}",
"passwd_tool_path": "htpasswd",
"passwd_file_path": "/etc/nginx/erigon_htpasswd",

Parameters for the script

Script behavior can be modified by the following environment variables:


default value




OS user the service is running on



Erigon data directory. Each chain data is collected in the subdirs


HAS TO be set before running the installation script, e.g.


Test this script without nagging letsencrypt services. If not set, (the inverse) ERIGON_USE_SSL is set


Not needed when ERIGON_DISABLE_SSL is set. Email address for the certbot to remind of SSL certificate renewal.

Running the script

  1. Download the installation script on the installation machine

    curl -sSfL -o \
  2. Review and edit the script if needed

  3. Execute the script

    Example script invocation (provider values for your environment)

    sudo env ERIGON_USER=golem \ \
    ERIGON_DATADIR=/data/erigon \

    During the installation you will be asked to:

    • Accept Golem's licence terms,

    • Provide the name for the provider node,

    • Provide the subnet the node will be subscribed to,

    • Provide your wallet address,

    • Provide the desired price per hour (in GLM).

  4. Make sure the port is opened

    It depends on how your machine was provisioned, but you might need also to allow ingress connections on the firewall.

    sudo ufw allow 8545/tcp

Check installation

After a successful installation, verify that the service is running.

sudo systemctl status golem

You can also check the logs of the running service

sudo journalctl -u golem -e