What can be done here guys???
Just some thoughts..........
The holder for the high speed spindle looks to be rigid enough.
I would leave it alone for now.
Check the Holder:
Put an indicator on holder plate to measure the force of pushing with your finger on the holder out at the high speed spindle and see any rotation of holder about the Z and do the same for the pushing upwards. It should take a fair amount of force to see a measurable deflection. Fix the deflection problem if it exists.
Check the spindle:
Check for any side and vertical play in the spindle. Should be no sideways
but you will probably have some vertical. I modified the hand / cable drive spindles I use. Needed to be carefull since there is expansion of the spindle as it warms at high speed. Some high end high speed spindles require that the spindle bearings be replaced after so much run time.
You may not be able to do any improvements and have what you have,but, t least you know that runout can accur if feeds, chip loads, loading, etc. can cause problems and need to adjust machining parameters.
Feed rate - Chip load etc:
First you may want to consider using a two flute high helix end mill rather than single flute as that can help with the chatter. STep over and cut depth will vary depending on material, feedrate etc. There are some guidelines to start but some experimenting is worth doing to find what works.
For quick range of machining paramters there is an end mill chart in Members Doc's or use some of the manufacturers on site calaculators for quidence.
Good flooding of the end mill to assist in chip removal is required.
Oscillation / resonance can be discussed ( it can get complex quickly)
,BUT, it the above is looked at you may find you don't that type of problem to begin with.
Finaly........
Yeah, you want to get things done efficiently / quickly as possible but if the basics are out of whack to begin with you will have problems.Everthing is just a trade off.
RICH