Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control

Designing optimal controllers for complex dynamical systems is hard when the system’s derivatives are discontinuous, unreliable, or completely missing. Gradient‑based reinforcement learning and classic optimal‑control methods often fail in these non‑smooth or black‑box situations. This project investigates Hamiltonian Primal‑Dual Policy Optimization (HPDPO), a derivative‑free, model‑based reinforcement‑learning framework for finite‑horizon optimal control. The approach treats control actions as a time‑indexed policy that spans the whole horizon. Instead of back‑propagating through the state dynamics or using a learned critic, HPDPO evaluates each candidate action by running deterministic suffix rollouts of the transition model. Future‑cost information is added through a regularized Hamiltonian dual formulation, which is solved with block‑coordinate multiplier updates. The paper

shows that HPDPO works well in several different settings, including: – Non‑smooth analytical problems – A learned ReLU biomarker cascade – An amyloid Neural ODE – Multi‑dimensional control tasks with up to five simultaneous controls For M.Tech students, the material offers practical advice on how to set up simulations that compare HPDPO with traditional gradient‑based controllers, how to assess trajectory‑tracking performance, and how to implement the block‑coordinate updates in Python or MATLAB.

Message: Hi FE, I need help with “Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control” in “Embedded Systems & Control”