This is a bug in After Effects on macOS 10.12 Sierra. It does not affect 10.11 or earlier versions of macOS.
What appears to be happening is that when you press the Return key to rename, the line return character is also being placed in the keyboard buffer. Because After Effects selects the name of the layer/footage/effect/item when you press Return to rename, the line return character is replacing the highlighted text.
This is not obvious, and worse, if that item is referenced by an expression, the expression will be updated with the line break. This causes the expression to fail since the line break happens in the name string, and you can’t fix it without removing the line break from the name, since the line break is now a _required_ part of the name. (!)
This bug is fixed in the upcoming release of After Effects. Until then, we recommend you either:
A) Right-click on the item and choose Rename, instead of using the Return key.
B) Immediately after pressing Return to rename, press the Delete (backspace) key to delete the line break character.
While I’m on the topic of new bugs introduced into After Effects by macOS 10.12 Sierra, here are the others we are aware of:
– In the View menu, the Show Guides and Clear Guides commands are labeled as “Show Grid” and “Lock Guides”. (The existing Lock Guides and Show Grid commands are present; this is only a labeling problem with the Show Guides and Clear Guides commands.) Clicking on the commands still works, although the Show Guides command toggle state doesn’t work properly; we recommend you use the keyboard shortcut (Command+;) or the same command in the Grids and Guides Options button at the bottom of the Composition panel. Also, at the top of the View menu, a superfluous Show Tab Bar command appears, which does not work. (The forced addition of this Show Tab Bar command is what is causing the problem with the other commands.) In the upcoming release of After Effects, these bugs are fixed.
– Scrolling to zoom in the Composition, Layer, and Footage viewer panels is extremely sensitive on macOS 10.12 Sierra with Apple touch devices, like Apple’s Magic Mouse and Magic Trackpad. This does not affect non-touch devices, like a mouse with a physical scroll wheel. We believe this is an OS-level problem, as many applications are reporting the same issue. We are continuing to investigate this, but have heard, anecdotally, that the 10.12.1 update (currently available as a beta) fixes this. (While investigating, we did take this opportunity to make a change: in the upcoming release of After Effects, inertia scrolling is now disabled when zooming in the viewer panels, which makes the scrolling experience much smoother on all versions of macOS.)