{
    "componentChunkName": "component---src-templates-note-note-js",
    "path": "/notes/PPO",
    "result": {"data":{"mdx":{"body":"var _excluded = [\"components\"];\nfunction _extends() { _extends = Object.assign ? Object.assign.bind() : function (target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i]; for (var key in source) { if (Object.prototype.hasOwnProperty.call(source, key)) { target[key] = source[key]; } } } return target; }; return _extends.apply(this, arguments); }\nfunction _objectWithoutProperties(source, excluded) { if (source == null) return {}; var target = _objectWithoutPropertiesLoose(source, excluded); var key, i; if (Object.getOwnPropertySymbols) { var sourceSymbolKeys = Object.getOwnPropertySymbols(source); for (i = 0; i < sourceSymbolKeys.length; i++) { key = sourceSymbolKeys[i]; if (excluded.indexOf(key) >= 0) continue; if (!Object.prototype.propertyIsEnumerable.call(source, key)) continue; target[key] = source[key]; } } return target; }\nfunction _objectWithoutPropertiesLoose(source, excluded) { if (source == null) return {}; var target = {}; var sourceKeys = Object.keys(source); var key, i; for (i = 0; i < sourceKeys.length; i++) { key = sourceKeys[i]; if (excluded.indexOf(key) >= 0) continue; target[key] = source[key]; } return target; }\n/* @jsxRuntime classic */\n/* @jsx mdx */\n\nvar _frontmatter = {\n  \"aliases\": [\"Proximal Policy Optimization\"]\n};\nvar layoutProps = {\n  _frontmatter: _frontmatter\n};\nvar MDXLayout = \"wrapper\";\nreturn function MDXContent(_ref) {\n  var components = _ref.components,\n    props = _objectWithoutProperties(_ref, _excluded);\n  return mdx(MDXLayout, _extends({}, layoutProps, props, {\n    components: components,\n    mdxType: \"MDXLayout\"\n  }), mdx(\"p\", null, \"(PPO) is a reinforcement learning algorithm that is widely used for training agents in environments where they need to learn \", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"/notes/optimal-policies\",\n    \"title\": \"optimal policies\"\n  }, \"optimal policies\"), \" through \", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"/notes/trial-and-error\",\n    \"title\": \"trial and error\"\n  }, \"trial and error\"), \". It is part of the family of \", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"/notes/policy-gradient-methods\",\n    \"title\": \"policy gradient methods\"\n  }, \"policy gradient methods\"), \", which directly optimize the policy that an agent uses to decide actions based on its observations.\"), mdx(\"h3\", null, \"Key Features of PPO:\"), mdx(\"ol\", null, mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Objective Function\"), \":\"), mdx(\"ul\", {\n    parentName: \"li\"\n  }, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"PPO uses a surrogate objective function that includes a \", mdx(\"a\", {\n    parentName: \"li\",\n    \"href\": \"/notes/clipped-probability-ratio\",\n    \"title\": \"clipped probability ratio\"\n  }, \"clipped probability ratio\"), \" to ensure that policy updates do not deviate too much from the current policy. This helps maintain stable and reliable learning.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"The objective function can be expressed as:\\n$$ L^{CLIP}(\\\\theta) = \\\\hat{\\\\mathbb{E}}_t \\\\left\", \"[ \\\\min \\\\left( r_t(\\\\theta) \\\\hat{A}_t, \\\\text{clip}(r_t(\\\\theta), 1 - \\\\epsilon, 1 + \\\\epsilon) \\\\hat{A}_t \\\\right) \\\\right]\", \" $$\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"Here, $$( r_t(\\\\theta))$$ is the probability ratio of the new policy to the old policy,$$(\\\\hat{A}_t )$$ is the advantage estimate, and \", \"(\", \" \\\\epsilon \", \")\", \" is a hyperparameter that controls the clipping range.\"))), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Clipping Mechanism\"), \":\"), mdx(\"ul\", {\n    parentName: \"li\"\n  }, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"The clipping mechanism prevents large updates to the policy, which can \", mdx(\"a\", {\n    parentName: \"li\",\n    \"href\": \"/notes/destabilize-learning\",\n    \"title\": \"destabilize learning\"\n  }, \"destabilize learning\"), \". By clipping the probability ratio, PPO ensures that the policy changes are small and controlled.\"))), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Advantages\"), \":\"), mdx(\"ul\", {\n    parentName: \"li\"\n  }, mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"Stability\"), \": The clipping mechanism provides a balance between \", mdx(\"a\", {\n    parentName: \"li\",\n    \"href\": \"/notes/exploration-and-exploitation\",\n    \"title\": \"exploration and exploitation\"\n  }, \"exploration and exploitation\"), \", leading to more stable training compared to other policy gradient methods.\"), mdx(\"li\", {\n    parentName: \"ul\"\n  }, mdx(\"strong\", {\n    parentName: \"li\"\n  }, \"Simplicity\"), \": PPO is relatively simple to implement and tune, making it a popular choice for many reinforcement learning tasks.\"))), mdx(\"li\", {\n    parentName: \"ol\"\n  }, mdx(\"p\", {\n    parentName: \"li\"\n  }, mdx(\"strong\", {\n    parentName: \"p\"\n  }, \"Applications\"), \":\"), mdx(\"ul\", {\n    parentName: \"li\"\n  }, mdx(\"li\", {\n    parentName: \"ul\"\n  }, \"PPO is used in a variety of domains, including robotics, game playing, and any scenario where an agent needs to learn complex behaviors through interaction with an environment.\")))), mdx(\"p\", null, \"Overall, PPO is valued for its effectiveness and ease of use, making it a go-to algorithm for many \", mdx(\"a\", {\n    parentName: \"p\",\n    \"href\": \"/notes/reinforcement-learning\",\n    \"title\": \"reinforcement learning\"\n  }, \"reinforcement learning\"), \" practitioners.\"));\n}\n;\nMDXContent.isMDXComponent = true;","inboundReferences":[{"id":"3ff3a715-cbe3-5679-8fb0-ece014496b25","slug":"KL-divergence"},{"id":"f5343dd1-9a13-55a7-93ea-0b68db08427f","slug":"machien-learning-optimisaiton-algorithms"}]}},"pageContext":{"slug":"PPO","node":{"childMdx":{"slug":"PPO"}}}},
    "staticQueryHashes": []}