想问个问题,falcon的注意力权重是fused_qkv的形式,用col_nn.Linear1D_Col这个函数进行切分是否正确???假设我的tp_size=2,会不会造成切分错误,为啥不用FusedLinear1D_Col #5222
Unanswered
laiqinghan
asked this question in
Community | Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
想问个问题,falcon的注意力权重是fused_qkv的形式,用col_nn.Linear1D_Col这个函数进行切分是否正确???假设我的tp_size=2,会不会造成切分错误,为啥不用FusedLinear1D_Col.这种切分方式在运行上应该不会报错,但是如果是微调huggingface的模型会不会造成计算错乱,直接对[QKV]切分,假设tp_size=2,会不会导致K被切分了,但Q,V并没有被切分,要是这样的话,下面这行拆分q,k,v的代码也可以跑通,但是计算逻辑上是否有问题呢???:祈祷:
![image_720](https://private-user-images.githubusercontent.com/22470281/294110386-9e1fae39-0381-45c7-ba0a-367690b88f53.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNjA5MTcsIm5iZiI6MTcyMTE2MDYxNywicGF0aCI6Ii8yMjQ3MDI4MS8yOTQxMTAzODYtOWUxZmFlMzktMDM4MS00NWM3LWJhMGEtMzY3NjkwYjg4ZjUzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE2VDIwMTAxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkzZGE2ZTMxM2IyMjMwZWE3MmYxZmZjMTg4NDFlNDNjNzQzM2M0OWVhZThhNWU4NjEwY2RjZjM4N2U3ZTBlNTQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.h48yc5VSIEkL9l4JCFROSvY22iwBvIClgnV8LIU_5Dk)
![image_720](https://private-user-images.githubusercontent.com/22470281/294110405-40b75f3d-3451-4bae-a90e-7d4883dcc39f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNjA5MTcsIm5iZiI6MTcyMTE2MDYxNywicGF0aCI6Ii8yMjQ3MDI4MS8yOTQxMTA0MDUtNDBiNzVmM2QtMzQ1MS00YmFlLWE5MGUtN2Q0ODgzZGNjMzlmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE2VDIwMTAxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhZjYxNDI5OTI5OWExMmMyMDdjNmQ0YWRmYzdkZTAyODRiZjE3ODAwYmFiYTBmNGY4OGY5NGM1Mzk0Yzc3Y2EmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.qRi029pD-jyAeqTy8dv7AVQSH_F8MJQ6GSGNI7oZVSM)
![image_720](https://private-user-images.githubusercontent.com/22470281/294110420-3c5daabf-b62a-45d0-80dc-a5130318d771.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjExNjA5MTcsIm5iZiI6MTcyMTE2MDYxNywicGF0aCI6Ii8yMjQ3MDI4MS8yOTQxMTA0MjAtM2M1ZGFhYmYtYjYyYS00NWQwLTgwZGMtYTUxMzAzMThkNzcxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE2VDIwMTAxN1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJmYjUyM2I0OWI2NzdkMDFkMDZhZDUzMzQ4YzAwNWVhMzRmNjZkZDZjYzZlY2MzODkzMzE4YTAwYTk2MTlmZmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.m-AIXa-AQXQ-u9KCzOlGUB7HTXHhuZERV8JIluAtA1Q)
Beta Was this translation helpful? Give feedback.
All reactions