[PROPOSAL]: refactor zero/gemini folder structure

[ver217]

Proposal

Current folder structure of zero/gemini is very confusing. The main components are implemented in different folders. E.g.

Component	Folder
Legacy Zero	zero/
Low level Zero	zero/
Gemini Hook	zero/utils
ColoInitContext	utils/model/colo_init_context
Chunk/Gemini Manager	gemini/
Zero Optimizer	nn/optimizer/zero_optimizer
ZeroDDP/GeminiDDP	nn/parallel
zero_wrapper	nn/parallel

We can reorganize the folder structure. E.g.

zero
| --- legacy # all legacy code related to legacy zero
| --- low_level # code related to low level zero optim
       | --- LowLevelZeroOptimizer
| --- gemini 
       | --- GeminiDDP
       | --- GeminiOptimizer
       | --- ColoInitContext

Roadmap

• Reorganize zero/gemini folder structure
• Update examples/applications
• Reorgnaize test folder structure

Self-service

• I'd be willing to do some initial work on this proposal myself.

[FrankLeeeee]

Can you convert this to discussion?

[FrankLeeeee]

Looks good to me.

[kurisusnowdeng]

I prefer naming the module after Gemini, while zero & chunk management are both its sub-functionalities

[SauravMaheshkar]

If I'm not wrong this proposal, has been worked through ?