Someone else can better explain this. Based on one of the video suggested in one...

rshm on May 18, 2023 | parent | context | favorite | on: Ask HN: Can someone ELI5 transformers and the “Att...

Someone else can better explain this. Based on one of the video suggested in one of the replies here. sin and cos doesn’t have any inherent properties specific to language, they were chosen because a simple linear function was needed in that step of optimization. Any other function could fit the bill as well.

vitno on May 18, 2023 [–]

Sin or cosine was chosen explicitly because they are non-linear functions