I use the term "marshaling" to mean "putting data/arguments in a form where they can pass through an interface that only accepts particular forms", whereas "serializing" specifically means "representing data with a byte stream". That is, serialization is one approach to marshaling, but also has uses outside the domain of what I would call marshaling.
Some examples of marshaling that aren't serialization:
- Putting arguments to a syscall in the registers that the kernel syscall entry point accepts. (I wouldn't call this serialization because there's no abstract order between the registers, and because some of the data might lie outside of the actual registers, being pointed to by them instead.)
- Building a tree structure from a JSON or XML source to pass to an interface that needs a tree. (This isn't serialization because the form is language-level objects with references between them, not a stream of bytes.)
- Passing data back and forth between a host application and an embedded language (like Lua).