rlNumericSpec

Create specifications object for a numeric action or observation channel

Description

An rlNumericSpec object contains specifications for a channel that carries an action or observation belonging to a continuous (infinite) set.

Creation

Syntax

spec = rlNumericSpec(dimension)

spec = rlNumericSpec(dimension,PropertyName=Value)

Description

spec = rlNumericSpec(dimension) creates a data specification object for a continuous action or observation channel and sets the Dimension property.

spec = rlNumericSpec(dimension,PropertyName=Value) creates the specification object spec and sets its Properties using one or more name-value arguments.

Properties

LowerLimit — Lower limit of the data space

-Inf (default) | scalar | matrix

Lower limit of the data space, specified as a scalar or matrix of the same size as the data space. When LowerLimit is specified as a scalar, rlNumericSpec applies it to all entries in the data space.

All agents and policies use this property to enforce lower limits on the observation and actions inputs. DDPG, TD3 and SAC agents and their policies use this property to enforce lower limits on the action. When using other agents, if you need to enforce constraints on the action, you must do so within the environment.

Example: LowerLimit=-1

UpperLimit — Upper limit of the data space

Inf (default) | scalar | matrix

Upper limit of the data space, specified as a scalar or matrix of the same size as the data space. When UpperLimit is specified as a scalar, rlNumericSpec applies it to all entries in the data space.

All agents and policies use this property to enforce upper limits on the observation and actions inputs. DDPG, TD3 and SAC agents and their policies use this property to enforce upper limits on the action. When using other agents, if you need to enforce constraints on the action, you must do so within the environment.

Example: UpperLimit=1

Name — Name of the rlNumericSpec object

string

Name of the rlNumericSpec object, specified as a string. Use this property to set a meaningful name for the signal carried by this data channel. This property is used by the RL Agent block to match the bus signal elements with their corresponding environment channels.

Example: Name="Observation"

Description — Description of the rlNumericSpec object

string

Description of the rlNumericSpec object, specified as a string. You can use this property to specify a meaningful description of the signal carried by this environment channel.

Example: Description="Measured Cart Velocity in m/s"

Dimension — Dimension of the channel

Read-only: vector

This property is read-only.

Dimension of the channel, specified as a vector. This property is the size of the element carried by the channel that carries data from an environment to an agent, and is essential for creating agents and function approximators objects that work with a given environment.

Example: Dimension=[3 1]

DataType — Information about the type of data

Read-only: "double" (default) | string

This property is read-only.

Information about the type of data, specified as a string, such as "double" or "single". The software uses this property to enforce data type consistency for observations and actions.

Example: DataType="double"

Object Functions

rlSimulinkEnv	Create environment object from a Simulink model already containing at least one agent block
rlFunctionEnv	Create custom reinforcement learning environment using your reset and step functions
rlValueFunction	Value function approximator object for reinforcement learning agents
rlQValueFunction	Q-Value function approximator with a continuous or discrete action space reinforcement learning agents
rlVectorQValueFunction	Vector Q-value function approximator with hybrid or discrete action space for reinforcement learning agents
rlContinuousDeterministicActor	Deterministic actor with a continuous action space for reinforcement learning agents
rlDiscreteCategoricalActor	Stochastic categorical actor with a discrete action space for reinforcement learning agents
rlContinuousGaussianActor	Stochastic Gaussian actor with a continuous action space for reinforcement learning agents

Examples

Create Specification Vector for Mixed Continuous-Discrete Observation Space

An rlNumericSpec object specifies an environment channel that carries signals (actions or observations) that belong to a continuous set. By contrast, an rlFiniteSetSpec object specifies a channel that carries signals that belong to a finite set (a set containing only a finite number of elements).

If you have an existing environment, you can extract its action or observation specifications (which in general are vectors of rlNumericSpec and rlFiniteSetSpec objects) using the getActionInfo or getObservationInfo functions. If, instead, you need to create a new custom environment, you must first define its action and observation channels. To do so, use rlNumericSpec or rlFiniteSetSpec.

For this example, you use both rlNumericSpec or rlFiniteSetSpec to define an observation space consisting of four channels, two of them continuous, and two discrete. To define the observation channel, use a vector of specification objects (each defining a single channel).

Here, the first observation channel carries a single number labeled 7, 9, 19 or -2. The second one carries a vector over a continuous three-dimensional space. The third channel carries a two by two matrix that can be either the zero matrix (that is, zeros(2)) or the identity matrix (that is, eye(2)). Finally, the fourth channel carries a continuous matrix with four rows and three columns.

obsInfo = [  rlFiniteSetSpec([7 9 19 -2])
             rlNumericSpec([3 1])
             rlFiniteSetSpec({zeros(2), eye(2)})
             rlNumericSpec([4 3]) ]

obsInfo=4×1 heterogeneous RLDataSpec (rlFiniteSetSpec, rlNumericSpec) array with properties:
    Name
    Description
    Dimension
    DataType

You can access each channel specification using dot notation.

obsInfo(1)

ans = 
  rlFiniteSetSpec with properties:

       Elements: [4×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

obsInfo(2).Name = "Velocity";
obsInfo(2).Description = "Velocity vector in m/s in body reference frame";
obsInfo(2)

ans = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: "Velocity"
    Description: "Velocity vector in m/s in body reference frame"
      Dimension: [3 1]
       DataType: "double"

You can now use these specifications to create both a new custom environment and an agent object that works within your environment.

Create Reinforcement Learning Environment from Simulink Model

For this example, you use rlNumericSpec to define an observation space consisting of a single channel carrying a three-element vector. You then use rlFiniteSetSpec to define an action space consisting of a single channel carrying only one of three possible values. You then use these observation and action specifications to create a custom Simulink® environment that relies on the rlSimplePendulumModel Simulink model.

The model represents a simple frictionless pendulum that initially hangs in a downward position. Open the model.

mdl = "rlSimplePendulumModel";
open_system(mdl)

An rlNumericSpec object specifies an environment channel that carries signals (actions or observations) that belong to a continuous set. By contrast, an rlFiniteSetSpec object specifies a channel that carries signals that belong to a finite set (a set containing only a finite number of elements).

If you have an existing environment, you can extract its action or observation specifications (which in general are vectors of rlNumericSpec and rlFiniteSetSpec objects) using the getActionInfo or getObservationInfo functions.

In this example, instead, you need to create a new custom environment. To do so you must first define the environment action and observation channels.

To define the channel that represents the observation space, use rlNumericSpec. The channel carries a vector containing three signals (the sine, cosine, and time derivative of the angle).

obsInfo = rlNumericSpec([3 1]) 

obsInfo = 
  rlNumericSpec with properties:

     LowerLimit: -Inf
     UpperLimit: Inf
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [3 1]
       DataType: "double"

To define the channel that represents the action space, use rlFiniteSetSpec. The channel carries a scalar expressing the torque and can be one of three possible values, -2 Nm, 0 Nm and 2 Nm.

actInfo = rlFiniteSetSpec([-2 0 2])

actInfo = 
  rlFiniteSetSpec with properties:

       Elements: [3×1 double]
           Name: [0×0 string]
    Description: [0×0 string]
      Dimension: [1 1]
       DataType: "double"

You can use dot notation to assign property values for the rlNumericSpec and rlFiniteSetSpec objects.

obsInfo.Name = "observations";
actInfo.Name = "torque";

You can now use these specifications to create both a new custom environment and an agent object that works within your environment.

To create your custom Simulink environment, use rlSimulinkEnv. Specify the Simulink model as first argument, the path of the agent block as a second argument, and the observation and action specifications that you have created in the previous step. For more information on custom Simulink environments, see Create Custom Simulink Environments.

agentBlk = mdl + "/RL Agent";
env = rlSimulinkEnv(mdl,agentBlk,obsInfo,actInfo)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : []
  UseFastRestart : on

Specify a reset function using dot notation. For this example, randomly initialize theta0 in the model workspace using the setVariable (Simulink) function.

env.ResetFcn = @(in) setVariable(in,"theta0",randn,"Workspace",mdl)

env = 
SimulinkEnvWithAgent with properties:

           Model : rlSimplePendulumModel
      AgentBlock : rlSimplePendulumModel/RL Agent
        ResetFcn : @(in)setVariable(in,"theta0",randn,"Workspace",mdl)
  UseFastRestart : on

Here, in is a Simulink.SimulationInput (Simulink) object, and the values of theta0 that you specify overrides the existing theta0 value in the model workspace for the duration of the simulation or training. The value of theta0 is then reverted to the original when the simulation or training completes. For more information on reset functions, see Reset Function for Simulink Environments.

You can now use env (together with an agent object) as argument for the built-in functions train and sim, which train and simulate the agent within the environment.

Version History

Introduced in R2019a