Online Methods vs Offline Methods

Online methods.

In RL, online methods refer to methods updating the model while interacting with the environment.

Pro

• Mathematically easy because most RL theories consider this method.

• Is usually much simpler than offline methods.

• Online methods are more stable.

Con

• Very data inefficient because each observation/state can be used once.

Offline methods.

In RL, offline methods refer to methods updating the model after having interacted with the environment.

Pro

• Data efficient, the same trajectory (history data) can be used for many updates.

Con

• Mathematically difficult because most RL theories are based on online methods.

• Is usually more complicated because it involves storing histories, re-weighting different history entries, and discarding old entries.

• Offline methods are less stable.