You may have a carbohydrate/sugar metabolic sensitivity. Feeling terrible after giving up Sugar is a classic symptom. Here's why:
The metabolism gets habituated to the Sugar rush, so insulin levels rise around the time you usually have Sugar in your tea or some other high glycemic index food. If you don't have your "fix" on schedule, you end up hypoglycemic.
Mild hypoglycemia can cause the shakes, headaches, dry mouth, etc. A severe case can cause shock, profuse sweating, confusion, disorientation, loss of balance, etc. In this case, one needs to sit down and have a small glass of fruit juice or something to raise blood sugar levels.
Check out the glycemic index and focus on eating foods with low glycemic load. Veggie and raw diets can be high in carbs and heavy on "high glycemic" foods. If you avoid the blood sugar highs and lows, it will be easier to control your weight.