yjdZddlmZmZmZddlZddlmZm Z m Z Gdde j Z Gdde j Z Gd d e j ZGd d e j ZGd de j ZGdde j ZGdde jZGdde jZGdde jZGdde jZGdde jZGdde jZdS)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc/,eZdZddejjdejjddddddddddddddddfdedede d e d ejd ed ejd e de de de de de de ede ede ede de e j de de de ede e de f.dZdS)TransformerEncoderSpecTFN' num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attention num_heads_kvhead_dim rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_basesliding_windowqk_normpre_post_layer_normc  | rdkrtdd| |_tjd||_||_tjd||_tjd||_dt|D|_ d|_ s st|_ |r|stj |_|rtj |_|,tjd ||_  f d t|D|_dS) a'Initializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of dimensions per attention head. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. sliding_window: Max sequence length to retain in KV Cache. qk_norm: Apply layer normalization to the query and key projections. pre_post_layer_norm: Add post layer norm for each pre norm layer. Nr 5Enabling multi_query_attention implies num_heads_kv=1int16int8c4g|]}tjS)rEmbeddingsSpec).0_s c/usr/local/lib/hermes-agent/venv/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py z3TransformerEncoderSpec.__init__..Ss.   -.K & ( (   Trint32cH g|]}t    S)) rrrrrrrrrrr r"r#)TransformerEncoderLayerSpec)r+r,rrrr#r"rrrr rrrrs r-r.z3TransformerEncoderSpec.__init__..`sa    ("3(?!)!%"3$7&;'$7      r/) ValueErrorrnpdtypetyperrrrrange embeddingsscale_embeddingsPositionEncoderSpecposition_encodingsr LayerNormSpec layer_normrr!layer)selfr rrrrrrrrrrrrrrrrrrr r!r"r#s ```` ``````` ``r-__init__zTransformerEncoderSpec.__init__ sx ! 'LA,=,= KL%:"'**// ::  (6**// ;; " 0 0 5 56F G G  278M2N2N   !%  <)@ <&9&;&;D #  KM K)7JJJDO  T'2'@('S'S'SD $  %"$(7"3"3"8"8"H"HD                 :&&!    r/)__name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolrrRotaryScalingTypefloatrAr)r/r-r r s #-8-C-H%&8C8S8Z$)"'(-&+&*"&$("&JN'("(,"'$)1f f f f  f  f  * f  #f &5f "f  f "&f f f  $f smf 3-!f "SM#f $ %f &&n&FG'f ( %)f *+f ,! -f .$/f 0"1f f f f f f r/r cLeZdZdejjdddddddddddddddddddddddddddddddddf#ded ed ed ejd ed ededededededededededededeededee j de de deded ed!ed"ed#ed$eed%eed&eed'eej d(eed)eed*ed+ed,eefJd-Z ed.ZdS)/TransformerDecoderSpecTFr Nr rr rrrrwith_encoder_attentionrproject_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibirrrrr original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normr#rrrr! quant_typequant_group_size quant_bitsr"v_norm external_pre_post_encoder_layersc&b #$%t|_r"|stdrtd|rdkrtddtjd||_||_tjd||_tjd| |_ tjd| |_ tj |_ d|_tj|_||_||_||_,tjd |_ s s|st-|_|r|stj |_|rtj |_tj|_% # $fd t;|D|_d |_tj|_ |p|k|jd <|r0tj|_!tj|_"| r | |jd<|"|jd<|!|jd<dSdS)a. Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) external_pre_post_encoder_layers: if the encoder attention pre and processing is done outside the attention. z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr r%r&r'Tr1r0cg|]K}tdidd d dd ddddd d d d d d dddd ddLS)rPrrrrrrrrr rWrXrYrZr#rrr!r"r^r_r))TransformerDecoderLayerSpec)r+r,r_rrrXrrWrYr#r"rrrr rrrrrZr!r^rPs r-r.z3TransformerDecoderSpec.__init__..sB   0/ (   '='= "3"3 )@(?     "   &:  #4"3 %8$7 '<&; (K 2R1Q )@(? #4"3 #4"3 %8$7  *\! ""# $ .~% & ' (v) *2R1Q+    r/Frquantization_typequantization_bitsquantization_group_size)#dict_configr4r5r6r7rrrrRrSrr*r9r:rOPTIONAL scale_outputsrTrUrVr!r;r<r=r>r LinearSpec projectionr8r?start_from_zero_embeddingfinal_logit_softcapping project_in project_out)&r@r rrrrrPrrQrrrRrSrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"r^r_s& ` `` `` `````````` ``` ```r-rAzTransformerDecoderSpec.__init__us3rvv  R T !RSSS% R !PQQQ  'LA,=,= KL'**// ::  (6**// ;;!x0055oFF!x0055oFF%466 $'0 ,H)&  %"$(7"3"3"8"8"H"HD ! <+ < <"&9&;&;D #  KM K)7JJJDO  T'2'@('S'S'SD $%022                        0:&&1    6*/&'1':$0E1 I % ,-  8)466DO*577D   G0:DL, -0:DL, -6FDL2 3 3 3 G Gr/c|jSN)rgr@s r-configzTransformerDecoderSpec.configs |r/)rBrCrDrrErFrIrJrrrKrL QuantizationrApropertyrsr)r/r-rNrNts -8-C-H$)'+#$"'(-! -2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(;@MhGhGhGhG hG  * hG " hG!%hGhGhG hG"&hGhGhGhGhG !hG"'+#hG$%hG&SM'hG( )hG*&n&FG+hG, %-hG./hG0+.1hG2"%3hG4 5hG6 7hG8"9hG: $;hG<sm=hG>3-?hG@! AhGB[56ChGD#3-EhGFSMGhGHIhGJKhGL+34.MhGhGhGhGTXr/rNc neZdZ d deededeejd ed ed ef d Z dS)r3FNTr r rrrrr r#ctjd|||||||| | | | |  |_t|||_|rt j||_t j||_t j||_ t j||_ t|jdt|jddSdS)NT) self_attentionrrrrrr!rrrrr r"glurr0r>) rMultiHeadAttentionSpecrxFeedForwardSpecffnrr=input_layer_normpost_attention_layer_normpre_feedforward_layer_normpost_feedforward_layer_normdelattr)r@rrrrrrr!rrrrr r"r#s r-rAz$TransformerEncoderLayerSpec.__init__%s "-C/$;%)!/ 3"7#   #wBBB  ,$/$=x$P$P$PD !-8-F!...D */:.G!///D +0;/H!000D , D' 6 6 6 DHl + + + + + , ,r/)FFFFNNNNTNr r FF) rBrCrDrrIrJrrKrLrAr)r/r-r3r3$s %$("&JN'("$)/,/,SM/, /,&n&FG/, %/,/,"/,/,/,/,/,/,r/r3c@eZdZ ddZdS) rbTFNr r rc tjdiddd|d|d|d|d|d|d | d | d | d | d |d|d|d|d||_|r!tj||||||du|_t |||_| ru|rt j|_n0t j|_ t j|_ t|jdt|jd|rt j||_ t j||_ |r6|r4t j||_ t j||_ t j||_t j||_t|jdt|jdt j|_dS)NrxTrrrrrrrr rWrXrrr!r"r^F)rrrr!r"has_normryr>r0r))rr{rx attentionr|r}rr=rZr~rr*external_post_encoder_attention_layer_norm)external_pre_encoder_attention_layer_normrrrrh layer_scalar)r@rPrrrrrrrrr rWrXrYrZr#rrr!r"r^r_s r-rAz$TransformerDecoderLayerSpec.__init__Xs0-C   4 // %<$; X  "z  0/  !4 3 #8"7 $  .N-M %<$; & X *> G 6! & " +B!)!-9UB DN#wBBB  ,  M)4)B)D)D&&(3(A(C(C%1<1J1L1L. D' 6 6 6 DHl + + +  ,$/$=x$P$P$PD !-8-F!...D *& *J -x@@@? -x@@@> /:.G!///D +0;/H!000D , D' 6 6 6 DHl + + +&/r/)TFFFFNTNr r rrFFFNNNFFFrBrCrDrAr)r/r-rbrbWso $ % )* !!).-[0[0[0[0[0[0r/rbceZdZddZdS)r|Fctj||_tj|_tj|_|rtj|_dSdS)Nr0)rr=r>rjlinear_0linear_1linear_0_noact)r@rzrs r-rAzFeedForwardSpec.__init__s`%3XFFF#.00 #.00  ;"-"8":":D    ; ;r/N)FFrr)r/r-r|r|s(;;;;;;r/r|ceZdZdZdS)r;c(tj|_dSrq)rrh encodingsrrs r-rAzPositionEncoderSpec.__init__s#,r/Nrr)r/r-r;r;s#-----r/r;c6eZdZdZddeeffd ZxZS)TransformerConfigz%Configuration for Transformer models.Nlayer_norm_epsilonc >tjdd|i|dS)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)superrAr@rkwargs __class__s r-rAzTransformerConfig.__init__. II,>I&IIIIIr/rqrBrCrD__doc__rrLrA __classcell__rs@r-rrs_//JJ8E?JJJJJJJJJJr/rc 6eZdZdZdedeffd Zedddej j dddej j dddddf d e eeeeffd ed ed ed edej dedededej dededededefdZedZedZdZdZdZxZS)TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecoderc>t|tstdt|tstdt ||_||_|j d|jj dS)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorrNrrArrrg add_attributer)r@rrrs r-rAzTransformerSpec.__init__s'#9:: QOPP P'#9:: QOPP P    "" #T\%G     r/FTrOr r rwith_relative_positionrrrrRrSrrrrrrrct|ttfr|\}}n||}}t|||||| | | || | || }t |||||| || ||| || }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) rrrrrrrrrrr) rrrrrrrRrSrrr)rlisttupler rN)clsr rrrrrrRrSrrrrrrrnum_encoder_layersnum_decoder_layersrrs r- from_configzTransformerSpec.from_configsV j4- 0 0 L5? 2  2 25? 2 (  '!"7- 34$;"7    )  '! 34$;++"7    s7G$$$r/cdS)Nrr)rrs r-namezTransformerSpec.name?s  r/cdS)Nr)rrs r-revisionzTransformerSpec.revisionCqr/ctSrq)rrrs r-get_default_configz"TransformerSpec.get_default_configGs """r/c.d|jjDS)Nc2g|]}|jjdS)r)weightshape)r+specs r-r.z>TransformerSpec.get_source_vocabulary_size..Ks"III !!$IIIr/)rr9rrs r-get_source_vocabulary_sizez*TransformerSpec.get_source_vocabulary_sizeJsII1HIIIIr/c:|jjjjdSNrrr9rrrrs r-get_target_vocabulary_sizez*TransformerSpec.get_target_vocabulary_sizeM|&-3A66r/)rBrCrDrr rNrA classmethodrrErFrGrHrrIrrJrrurrrrrrrs@r-rrs  - 8N      * (-#-8-C-H! %&8C8S8Z$)(-&+!O%O%#uS#X./O%O%!% O%  O%  O% *O%O%O% #O%&5O%"O%"&O%O%O% $!O%O%O%[O%b!!X!X###JJJ7777777r/rc6eZdZdZddeeffd ZxZS)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.Nrc >tjdd|i|dS)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)rrs r-rAz&TransformerDecoderModelConfig.__init__Trr/rqrrs@r-rrQ_77JJ8E?JJJJJJJJJJr/rcBeZdZdZdeffd Zedejj ddddddddddddddd d ddddddddddddfd e d e d e d ejde de de de de de de de de de e de de e jdedede de de de d e d!e d"e e d#e e d$e e d%e ejd&e e d'e e d(e d)e f@d*Zed+Zed,Zd-Zd.ZxZS)/TransformerDecoderModelSpecz3Describes a Transformer decoder model (e.g. GPT-2).rct|tstdt||_|jjD] \}}|j ||!dS)z|Initializes a Transformer decoder model specification. Args: decoder: The decoder specification. rN) rrNrrrArrsitemsrgr)r@rkeyvaluers r-rAz$TransformerDecoderModelSpec.__init__as '#9:: QOPP P  ,-3355 3 3JC L & &sE 2 2 2 2 3 3r/TFNr r rr rrrrrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"r^c!t||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d | }!||!S)!a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) rrrrPFrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"r^)rN)"rr rrrrrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"r^rs" r-rz'TransformerDecoderModelSpec.from_configosZ)  " " " X" "z " !4 3 " $)5 " (-" *>" 54" G" X" %" *F)E" $ " "z" 0/!" "!4 3#" $#8"7%" &$ '" (.N-M)" *%<$;+" ,0/-" .0//" 0!4 31" 2#8"73" 4&5" 6X7" 8*>9" :"z;" <.-=" >"z?" @GA" B6C" Hs7||r/cdS)NrNr)rrs r-rz TransformerDecoderModelSpec.name''r/cdS)Nr)rrs r-rz$TransformerDecoderModelSpec.revisionrr/ctSrq)rrrs r-rz.TransformerDecoderModelSpec.get_default_config,...r/c:|jjjjdSrrrrs r-get_vocabulary_sizez/TransformerDecoderModelSpec.get_vocabulary_sizerr/)rBrCrDrrNrArrrErFrIrJrrrKrLrtrrurrrrrrs@r-rr^s== 3 6 3 3 3 3 3 3 -8-C-H$)#$',-2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(Cpppp p  * p " ppp!%pppp'+ppSMp  !p"&n&FG#p$ %%p&'p(+.)p*"%+p, -p. /p0"1p2 $3p4sm5p63-7p8! 9p:[56;p<#3-=p>SM?p@ApBCppp[pd((X(X///7777777r/rc6eZdZdZddeeffd ZxZS)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.Nrc >tjdd|i|dS)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)rrs r-rAz&TransformerEncoderModelConfig.__init__rr/rqrrs@r-rrrr/rceZdZdZdejjfdededejffd Z e dZ e dZ d Z d ZxZS) TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcxt|tstdt||_|jd|jj|rFtj |_ tj d||_dSdS)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrr'N)rr rrrArrgrrrrj pooler_denser5r6r7pooler_activation)r@rrrrs r-rAz$TransformerEncoderModelSpec.__init__s'#9:: QOPP P   "" #T\%G     O + 6 8 8D %'Xf%5%5%:%:;M%N%ND " " " O Or/cdS)Nr r)rrs r-rz TransformerEncoderModelSpec.namerr/cdS)Nr r)rrs r-rz$TransformerEncoderModelSpec.revisionrr/ctSrq)rrrs r-rz.TransformerEncoderModelSpec.get_default_config#rr/cF|jjdjjdSr)rr9rrrrs r-rz/TransformerEncoderModelSpec.get_vocabulary_size&s|&q)06q99r/)rBrCrDrrrETanhr rJrArurrrrrrs@r-rrs<< $5@5K5P OO'OO(2 OOOOOO4((X(X///:::::::r/r)rtypingrrrnumpyr5ctranslate2.specsrrr LayerSpecr rNr3rbr|r;SequenceToSequenceModelConfigrSequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrr)r/r-rs66))))))))))EEEEEEEEEEg g g g g Z1g g g TmmmmmZ1mmm`0,0,0,0,0,*"60,0,0,f\0\0\0\0\0*"6\0\0\0~;;;;;j*;;;-----*.--- J J J J J @ J J J}7}7}7}7}7j<}7}7}7@ J J J J JJ$B J J JP7P7P7P7P7*">P7P7P7f J J J J JJ$B J J J):):):):):*">):):):):):r/