{"id":481,"date":"2022-10-31T21:48:27","date_gmt":"2022-10-31T13:48:27","guid":{"rendered":"https:\/\/scutvk.cn\/?p=481"},"modified":"2022-10-31T21:48:54","modified_gmt":"2022-10-31T13:48:54","slug":"get-all-layers-hidden-states-of-debertahuggingface","status":"publish","type":"post","link":"https:\/\/scutvk.cn\/?p=481","title":{"rendered":"Get all layers hidden states of deberta(Huggingface)."},"content":{"rendered":"\n<p>Usually, we use deberta like:<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>all_plm_output = self.plm(document_batch&#91;doc_id]&#91;:self.plm_batch_size, 0],  # &#91;1, 512]\n                                      token_type_ids=document_batch&#91;doc_id]&#91;:self.plm_batch_size, 1],\n                                      attention_mask=document_batch&#91;doc_id]&#91;:self.plm_batch_size, 2])\n# self.plm is a pretrained deberta-v3-base<\/code><\/pre>\n\n\n\n<p>And we can get the output as follows:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"153\" src=\"https:\/\/x2.mday.top\/wp-content\/uploads\/2022\/10\/image-1024x153.png\" alt=\"\" class=\"wp-image-482\" srcset=\"https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1024x153.png 1024w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-300x45.png 300w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-768x115.png 768w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-769x115.png 769w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image.png 1262w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption>It only has &#8220;last_hidden_states&#8221;.<\/figcaption><\/figure>\n\n\n\n<p>To get all the hidden layers&#8217; output, I check the docs of deberta in huggingface.Click here to docs: <a href=\"https:\/\/huggingface.co\/docs\/transformers\/main\/en\/model_doc\/deberta#transformers.DebertaPreTrainedModel\" target=\"_blank\" rel=\"noopener\">DeBERTa (huggingface.co)<\/a><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"708\" src=\"https:\/\/x2.mday.top\/wp-content\/uploads\/2022\/10\/image-1-1024x708.png\" alt=\"\" class=\"wp-image-483\" srcset=\"https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1-1024x708.png 1024w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1-300x207.png 300w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1-768x531.png 768w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1-769x532.png 769w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1.png 1215w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>forward() has a argument named &#8220;output_hidden_states&#8221;, which set to &#8220;True&#8221; can output all hidden states.<\/p>\n\n\n\n<p>code:<\/p>\n\n\n\n<pre class=\"wp-block-code has-small-font-size\"><code>all_plm_output = self.plm(document_batch&#91;doc_id]&#91;:self.plm_batch_size, 0],  # &#91;1, 512]\n                                      token_type_ids=document_batch&#91;doc_id]&#91;:self.plm_batch_size, 1],\n                                      attention_mask=document_batch&#91;doc_id]&#91;:self.plm_batch_size, 2],\n                                      output_hidden_states=True)<\/code><\/pre>\n\n\n\n<p>result:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"1024\" height=\"210\" src=\"https:\/\/x2.mday.top\/wp-content\/uploads\/2022\/10\/image-2-1024x210.png\" alt=\"\" class=\"wp-image-484\" srcset=\"https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-2-1024x210.png 1024w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-2-300x62.png 300w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-2-768x158.png 768w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-2-769x158.png 769w, https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-2.png 1087w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>We can get all hidden states in &#8220;hidden_states&#8221; now.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Usually, we use deberta l&hellip;<\/p>\n","protected":false},"author":1,"featured_media":483,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[10,12,2],"tags":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/scutvk.cn\/wp-content\/uploads\/2022\/10\/image-1.png","_links":{"self":[{"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/posts\/481"}],"collection":[{"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scutvk.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=481"}],"version-history":[{"count":1,"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/posts\/481\/revisions"}],"predecessor-version":[{"id":485,"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/posts\/481\/revisions\/485"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scutvk.cn\/index.php?rest_route=\/wp\/v2\/media\/483"}],"wp:attachment":[{"href":"https:\/\/scutvk.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=481"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scutvk.cn\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=481"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scutvk.cn\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=481"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}