​
Environment

An Environment in the HUD SDK represents a running instance where an agent can interact with a task. It provides methods for observation, action, and built-in telemetry to record the agent’s interactions.

​
Overview

Environments are the core runtime instances where agents perform tasks. Key features include:

  • Gymnasium-compatible interface for agent interaction
  • Built-in telemetry to record actions and observations
  • Methods to evaluate agent performance
  • Support for video generation of agent interactions
  • Automatic trajectory creation for later analysis

​
Initialization

Environments are created using the gym module:

import hud
from hud import gym

# Create a job to group related trajectories
job = await hud.Job.create(name="example-job")

# Create an environment instance
env = gym.make("OSWorld-Ubuntu", job_id=job.id)

# Wait for environment to be ready
await env.wait_for_ready()

# Load a task and reset the environment
task = await hud.Task.get("osworld_task_id")
observation = await env.reset(task_id=task.id)

​
Key Methods

The Environment class provides several key methods:

​
Reset

Resets the environment to a specific task:

observation = await env.reset(task_id="task-123", metadata={"agent_version": "1.0"})

​
Step

Takes an action in the environment and returns the new state:

observation, reward, terminated, info = await env.step(action)

​
Evaluate

Evaluates the current task:

result = await env.evaluate()

​
Get Trajectory

Retrieves the trajectory record of the agent’s interactions:

trajectory = await env.get_trajectory()
video_url = await trajectory.get_video()

​
Close

Closes the environment:

await env.close()

​
Observations

Observations from the environment include:

  • screenshot: A base64-encoded PNG image of the current screen
  • text: Text observation from the environment
  • state: Additional state information specific to the environment type

​
Environment States

An environment can be in one of several states:

  • creating: The environment is being created
  • running: The environment is running and ready for interaction
  • error: An error occurred during environment creation or execution
  • closed: The environment has been closed

​
VNC Access

For debugging purposes, you can view the environment directly via VNC:

live_url = await env.get_vnc_url()
print(f"Connect to the environment via VNC at: {live_url}")
  • Task - The configuration that defines what the agent needs to accomplish
  • Trajectory - The record of agent interactions in the environment
  • Job - Groups of related trajectories across multiple tasks