The orbs are ball-like spheres containing the memories of the person they belong to. The Orbs differentiate in color, each one representing a specific emotion - yellow for Joy, blue for Sadness, green for Disgust, purple for Fear, and red for Anger. They are activated by the Emotions in Headquarters of a person to make that person remember things.
During the events of Inside Out, after Riley moves to San Francisco, if Sadness touches a Memory Orb, the memory will become sad and the orb's color will change to blue, regardless of original color. This is most likely because the affected memories are all from Minnesota and Riley is not going back, which makes those memories sad. If this hypothesis is correct, presumably nothing would happen to the San Francisco-origin memories, since they were created at her new home.
At the end of the movie, Joy and Sadness combining forces to bring Riley home reveals that two emotions can create a memory. Afterwards, more emotions can take control at once with their new, larger console, and two-color memories become common.
As memories age, their video playback becomes blurry, the audio becomes muffled, and their color gradually fades. The Forgetters remove the older memories from Long Term Memory and throw them in the Memory Dump when this happens, where the memories will eventually crumble to dust and blow away. If a memory is removed by a Forgetter, it appears to take on a brown color when it goes into the dump, which might originate from the color of old photo film.
Some memories, which contain only factual information (like phone numbers or times tables), seem to have no initial color at all and are simply gray to show they are neutral.