Program Strategic Planning and Design

[IDMclean]

I've taken a few classes in programming and I understand some major concepts. The one that I am currently most focused on is the design and development cycle of a application. I am also rather interested in multi-language programs and programming hooks.

 

Here's the subject I would like to know more about. How can one use things like UML, XML, and flowcharting to design a program? More generally, how does one go about planning a small software development project?

 

I want to write an application to generate a world-wide culture complete with sub-cultures. It must give demographics, economic and industrial information, city compositions, organizations, nationalities, and type of government. I would like it to use a GUI and write the information into accessible Files.

 

Feel free to ask questions, request clarifications or specifications, and to ask for more information. I will just ask one thing: keep it descriptively detailed (specific).

[Buffy]

...The one that I am currently most focused on is the design and development cycle of a application....

This was big back in the 70s and 80s: Building a complete formalism to the process of program development fell under the name Computer-Aided Software Engineering or "CASE". These tools mostly had three (and sometimes four) pieces:

• Requirements definition: Mostly english language descriptions of what you wanted to accomplish, with some definition of relationships between the key elements. Good mostly for identifying conflicts in requirements, and producing documentation showing that the IT group was actually doing something to make sure the end users were going to be happy.
  
• Database design tools: let you visually design your database elements and their relationships. Pretty much essential to keeping track of database definitions, and they're pretty much integrated into every database tool today.
  
• Dataflow Diagrams or Process definition: This is trying to get the definition of what the program is supposed to do down in detail. One of the first things you find out is that if you're doing anything really complicated, you've got layers and layers of traditional "flow-charts" (process definition) and you have to start thinking in terms of "data-flow" which is how the high-level processes cause the data to move and be manipulated through the course of the application.
  
• Code-generation: Most CASE tools generate database definition code because its pretty straightforward. Code-generation from process definitions and data-flow diagrams is...uh...totally hard. So what most of these systems do is simply say "write the code that goes in this box in your diagram": its just providing a structure rather than actually generating your code. This is actually useful and a lot of theory and tools have been built up around them, some turning into complete cults (look up Grady Booch)...

Not popular now (see below) but they're something that you'd be advised to study.

I am also rather interested in multi-language programs and programming hooks.

I'm a firm believer in using the right tool for the right problem. Sometimes implementations will conspire against you though, so this is an area that is ripe for tradeoffs: Lisp is great for certain AI problems, but if you want it to run fast, you have to go compiled and throw out using some of its cooler features like dynamic execution.

Here's the subject I would like to know more about. How can one use things like UML, XML, and flowcharting to design a program?

UML and XML can be used to store "flowcharting" data of course, and most newer tools for program design do this, but its just a data storage mechanism, and the only benefits are potential interoperability (which in most cases doesn't exist because XML just makes it easy to read and you still have the semantic problem of interpretation and application of the data).

More generally, how does one go about planning a small software development project?

Very carefully. :hihi:

 

No, seriously, in spite of the popularity of neo-Rapid Application Development (first popular during the backlash against CASE back in the 80s, but recently revived with "Agile Development" and other fads), its really important to carefully map out your requirements and do a design before writing a line of code or even defining your data. Then realize that the tendency to want to not throw out early parts of the design once implementation problems crop up kills projects, so you have to plan for redesign along the way (my own company's program is in the middle of its third *complete rewrite*).

 

Software development is not simple, and there are ongoing complaints that its not at all disciplined (I'll be the first to agree with that), and that it would benefit from a stronger, more traditional Engineering approach than is common today.

 

Remember Justin Case: the world-class coder of well-defined bugs,

Buffy

[alexander]

i hate typing monkeys: coders that just sit and code without planning. Although i hate doing flowcharting, and generally plan little on how i structure the whole project, i very oftenly spend many hours thinking through algorithms to come up with better then average ways of doing things. I mean generally on larger projects, you would spend lots of time designing the most efficient ways to code a particular piece, and design is crucial in the Object Oriented Programming world as in the end you know exactly the classes methods and inherritance you will use, and that is a good 60% of what you need to know to write a better piece of code in most cases. Unfortunately since i dont have the luxury of time, many times i write quick and sometimes dirty code, but hey sometimes you have deadlines that have to be met, and you just don' care enough for the cleanliness.

 

shoot gotta run to class, i'll post more tomorrow or later on if i have time...

[IDMclean]

I should make an addendum, I have in mind to present this design to a number of programmers and I, myself, will be keeping my hands to myself. The design would do well to speak for itself.

[LaurieAG]

More generally, how does one go about planning a small software development project?

 

I want to write an application to generate a world-wide culture complete with sub-cultures. It must give demographics, economic and industrial information, city compositions, organizations, nationalities, and type of government. I would like it to use a GUI and write the information into accessible Files.

 

Hi KickAssClown,

 

While Buffy has mentioned CASE there are many other systems development lifecycles such as REDI (Requirements, Evaluation, Development and Implementation).

 

While the modelling technique used usually depends on the type of material and information available to the designer/developer, in your case it would be beneficial to determine what your desired specific program outputs are and then work backwards to develop a data model (and internal/external procedures) that will provide your desired (and specified) outputs.

 

This is not that much different from analysing an existing system from its outputs (but you must create the outputs first).

 

I should make an addendum, I have in mind to present this design to a number of programmers and I, myself, will be keeping my hands to myself.

 

Provide a Context diagram, top and lower level DFD's (data flow diagrams) and an ER (entity relationship) diagram for your data structure and you'll probably find that any programmer (worth their salt) will find it a piece of cake (if you have the right model that is).

[IDMclean]

I return with greater experience!

 

I've recently been playing with Java 6, and it's many interesting features. I've started to tie my concept here with The Plan because I've discovered that Java now has the capability to reflect on itself.

 

Amongst it's features are:

java.lang.reflect

Generics

XMLEncoding/Decoding

Autoboxing

 

I've been reading about things like RDF/XML, the semantic web, debugging/bug databases, and CVS/SVN.

 

What do the programming experts think about leveraging these technologies to make the program manage it's formal design with user oversight?

[Buffy]

I return with greater experience!

Congratulations! :)

What do the programming experts think about leveraging these technologies to make the program manage it's formal design with user oversight?

Self-modifying code is tremendously cool, and I've always been convinced that it's where we have to go to truly advance software development.

 

Unfortunately, it's always been a bit like fusion power: forever "commercial in 20 years."

 

And I think you've hit the nail on the head in this statement: the key problems are "formal design" and "user oversight":

• How can you "formally design" something that is programming itself?
  
• When a computer's ability to generate data--or in this case code--so outstrips the human ability to absorb, how can there be any "oversight"
  

I think this is the very reason we have not gotten very far with this stuff. It's incredibly useful in "fuzzy" applications like AI, but the inability to "know" what the program is going to do means that most folks don't even know what to do with it. We're still at a level of software design where the notion that the goal of creating software should be to be able to provide a "proof" of it's operation: this thread discussing Edsgar Djikstra's complaints about CS education popped up this week on Slashdot. The lecture linked in the thread is a great read by itself, but what was interesting to me in seeing your post is that while he's quite correct about the fact that CS education was 20 years ago--and to a great extent still is, which is the content of the thread--short-sighted and in adequate, what we have yet to understand about CS may be off in a very different direction from what Edsgar was complaining about. CS is so much bigger than what we already know, and this notion of code generation (along with really getting a handle on developing multi-threaded apps) is a big part of it.

 

Keep digging!

 

I am looking for a lot of men who have an infinite capacity to not know what can't be done, :)

Buffy

[IDMclean]

[...]

 

And I think you've hit the nail on the head in this statement: the key problems are "formal design" and "user oversight":

• How can you "formally design" something that is programming itself?
  
• When a computer's ability to generate data--or in this case code--so outstrips the human ability to absorb, how can there be any "oversight"
  

I think this is the very reason we have not gotten very far with this stuff. It's incredibly useful in "fuzzy" applications like AI, but the inability to "know" what the program is going to do means that most folks don't even know what to do with it. [....]

 

Keep digging!

 

I am looking for a lot of men who have an infinite capacity to not know what can't be done, :phones:

Buffy

 

To the first, we can build it's seed/DNA. It what I intend to do.

To the second, we'll worry about that when it happens. For now, I know any sort of framework like what I have in mind is going to require human interaction to work.

 

In the attached, the framework uses an XML and RDF/XML description of the program, solutions, and problems. The inspector deals with checking the solutions (sub-routines/binaries) against a battery of problems (tests). The builder deals with object serialization, marshalling, demarshalling, and binary to representation rendering. Version control deals with interfacing the framework with a version control system, so that in the event of failure the framework can roll itself back to a previous working build. The bug/debugging database control allows the framework and the version control to register and retrieve failures, exceptions, errors, and unexpected function with the database to prevent updating to a broken branch/solution-set.

 

The Model-View-Controller is a well document design pattern. In my crazy brain, I see the Model being the diagram or language(s) itself, but for more practical applications it will be the thing which the program ultimately deals with.

 

Self-education is, I firmly believe, the only kind of education there is.

-Ian

[alexander]

Lets start with this, in my personal experience, educations is not about learning the specifics of a particular subject, for you can not, in the time educators have, teach someone any high level of proficiency in a subject, any given subject. Education is about enthusing one about particular topics of the subject, and encouraging them to study the subject on their own to gain their own level proficiency in a specific topic of the subject matter.

 

The bug/debugging database control allows the framework and the version control to register and retrieve failures, exceptions, errors, and unexpected function with the database to prevent updating to a broken branch/solution-set.

 

I recall reading a report on a genetic evolution virtualization project, i think it was UCal or something like that, they wrote a piece if self-modifying code... program that wrote programs (children). the thing was they added a thing to this that was able to sometimes randomly mutate code, parts of code... they ran like a 1000 generations, and reviewed the resilts (many hours of reading and testing code). From a base parent, that was able to do basic math operations, these 1000 generations children (not all, by far) were able to do rather complex math. some could multiply, divide, exponentiate,etc. the thing was, once they isolated the successful children, they traced their origins through the code trees. There were generations that derived code from parents that were incapable of doing anything useful, yet it was those mutations that enabled the later children to do amazingly complex math in the later generations... What i am saying is that not every bit of bad code should be gotten rid of, its those sometimes useless logic loops that will be able to leap the progress of your code in the future generations of your code :eek:

[CraigD]

IMHO, there’s a lot of advantage to writing self-modifying code that runs under interpreters for programming languages that intrinsically allow self-modifying code. Several standard languages (eg: LISP, MUMPS) fit this description.

 

Here are several small self-modifying programs in MUMPS. A short MUMPS primer is necessary for people unfamiliar with the language:

r[ead] lgvn : Accepts input from the current device (ie: stdin), assigning it to the variable (in MUMPS terms, a local or global node) named lgvn

f[or] : a flow control structure keyword. After code following it is executed, execution resumes from first character following it

x[ecute] expr : evaluates expression expr and executes it as MUMPS code

s[et] expr1=expr : evaluates expr and assigns it to a variable described by expr1

$p[iece](lgvn,delmtr,expr) : evaluates expr and lgvn and returns. Eg: if A="a+b+c", $p(A,"+",2)="b".

w[rite] expr : evaluates expr and outputs value to current device.

r X f  x X
w 1 s $p(X," ",2)=$p(X," ",2)+1

This program outputs the positive integers by modifying itself with each iteration to write a value one greater than before.

 

The following program requires one more chapter of MUMPS primer:

$tr[anslate](expr,expr2,expr3) : changes each character in expression expr matching a character in expr2 with the character in the corresponding position in expr3. Eg: $tr("testu2","aeiou","1234")=”t2st2”

r X f  x X
w 1 s $p(X," ",2)=$p(X," ",2)+2,X=$tr(X,"+*","*+")

This program outputs a slightly more peculiar sequence of integers (1 3 6 8 16 18 36 38 76 78 156) by modifying itself to alternate between modifying itself to output a value two greater than and twice that of before.

 

The following requires a few more chapter of MUMPS primer:

$et[rap] : code that is immediately executed when an error occurs in execution

$ec[ode] : value describing an execution error. If "" (the zero length string), execution continues. Otherwise, $et is executed as if it appear immediately following the current command.

f[or] lgvn=expr1:expr2:expr3 : as f[or], but assigns value of expr1 to variable named lgvn and increments it by value of expr2 until it exceeds value of expr3, then exits loop. (ie: same as C for (lgvn=expr1; lgvn<=expr3; lgvn+=expr2) )

$r[andom](expr) : returns a random integer between 0 and expr-1

$c[haracter](expr) : returns the ASCII character specified by the integer part of expr

$g[et](lgvn[,expr]) : returns value of variable named, lgvn or, if lgvn is undefined, value of expr is returned, or, if expr not present, "".

:expr : when appended to a command keyword, command execution occurs only if expr evaluates to a non-zero number.

s $et="s $ec=""""" f  s X="" x "f I=1:1:$r(250) s X=X_$c($r(96)+32)",$g(X1),X

This program just executes random strings in a “million monkeys” manner. The program it can create is limited in size only by allocated disk space of the MUMPS system in which it runs. Eventually, it might create and execute a program for an interactive conversation with a simulation of Shakespeare. Vastly more likely, it will first stumble upon the MUMPS h[alt] command and end its execution, or less likely but more likely than channeling Shakespeare, do something dreadful.

 

Certain kinds of games of the “have rules to change their rules” varieties (eg: Magic, some old puzzles, cheap knockoff Harry Potter boardgames) can be coded more easily with careful use of self-modification. I’ve flirted with more interesting uses of self modification. The latest version of my GRAVSIM* spacecraft simulator found in the Space forum uses something of the sort to allows quick changes to spacecraft design and addition of external conditions. I want to take some serious time and write a agent-based simulator in a vaguely physical world-like simulated environment – I’m leaning toward one of my favorite movie genres, Romero-esque zombie survival horror – but free time is so hard to find! :eek: