Performing unauthorized work in waters of the United States or failure to comply with the terms of a valid permit can have serious consequences. You would be in violation of Federal law and could face stiff penalties, including fines and/or requirements to restore the area.
Enforcement is an important part of the Corps regulatory program. Corps surveillance and monitoring activities are often aided by various agencies, groups, and individuals, who report suspected violations. When in doubt as to whether a planned activity needs a permit, contact the nearest district regulatory office. It could save a lot of unnecessary trouble later.