Here are questions that you should answer before you start to code an OS. It's not a secret that every program must be designed before it is written (implemented). Also it's not a secret that a very few of programs are written this way... Most are written under pressure and hardly have a complete requirements list when coding starts.
Such an approach is very harmful. Design is finished in breaks between coding work and is dictated by things that are already written and work. Programmers tend to think "I'll change it later" but mostly later never comes. Nobody really wants to change already working code to fit a changed design.
It may be very painful if you write an OS. Because of design flaws or incomplete design the whole project may be brought down or at best take months to redesign and rewrite big parts.
Answering questions suggested here will not form a full design specification. However it will not let coding go completely off road. All of them should be answered before writing a line of code. Certain questions may be ignored if they are irrelevant, e.g. you don't need to think about file systems if you don't plan to support mass storage devices. From the other hand, all the answered questions should not contradict. Where I feel necessary, hints are typed as italic.
What is a primary goal of my OS?
Is it a standard (low end) desktop system? User is dummy, highest priority for hardware and software compatibility.
Is it a high-end desktop system? User is CAD/CAM engineer, highest priority for performance and certain hardware/software compatibility.
Is it a real-time oriented system? User is a professional programmer, highest priority for performance, defined response time, easy extendable hardware support and programming control.
What platforms my OS is going to support?
Will it support multiprocessing?
What kind of multiprocessor platforms? Symmetric? (all processors are exactly the same). Asymmetric? (CPUs may be different in architecture and computing power). Both?
Will it support only local multiprocessing? (all CPUs are connected through a local bus). Distributed multiprocessing? (CPUs are connected through network-like connection). Both?
What is the target hardware system? Desktop? (more or less standard hardware set). Customizable (embedded) hardware? (If the latter is an answer you'll likely have to individually support every even compatible processor).
Will it be a multitasking OS?
What kind of multitasking will it provide for applications? Cooperative? (tasks yield CPU when they don't need it, demonstrating good will). Preemptive? (tasks are given a defined amount of CPU time).
Do I need to protect tasks from each other well?
What is a relationship between tasks in terms of living space? Do they share the same address space? Completely separated? Both?
How will different tasks communicate with each other?
What will be a memory model of space that a task runs in? Should I favor simplicity and speed (memory is cheap) or size (memory is a scarce resource)?
Do I need to protect system from application tasks?
What file system will my OS use?
Should I favor access time (performance) or reduced storage space (size)?
Can I use one of already developed and well documented file systems?
Can I use a cut down version of one of well-known file systems?
What will be an executable format?
What build tools do I need?
Can I use one of existent compilers and linkers?
Can I obtain (for free, buy or lease) source code for compilers and linkers?
Do I have to write my own several tools?
Do I have to write all tools on my own?This should be by any means avoided.
How can I easily support third party soft?
Can I support already existent and popular software?
How can I support easy creating of third party applications for my OS? (Libraries)
How can I support easy creating of third party device drivers?
How can I use already written code and information?
Can I use code that is written by others and works? (Even partially).
Where can I get different kinds of information? (Set your own information library).
