These two are often mixed up in speech. If you are doing something in a formal manner, you are behaving formally; but if you previously behaved differently, you did so formerly.
List of errors
