Programming Challenge!

Programming Challenge! [View] [Reply] [Top]
Posted by Delta Lt dopel On 2008-03-06 17:26:59

Since this forum is dedicated to teaching people how to program I figured it might be a good idea to have a task. First off, this task is not programming 101. It will touch on using data structures, socket programming, and some coding in more than one language. To that end, the goal is to learn. So I will be providing my solution as separate posts if you wish to use them as a guide. This task is a real scenario starting from how it came to me and concludes with a final implemented solution. To be honest, I am not yet complete, so I will update the stages as I get the time. While I am not quite finished with the code, I have worked out my plan and finished a proof of concept (prototype). I am in the stage of tuning up a fully working model. This will allow anyone interested some time to work and most importantly, think, before being spoiled with a solution. I will post this in a few stages: the problem, solution, and implementation. Each phase will be a separate child post of this thread. You may post questions and or your own comments and solutions. As this is a real world problem, the only incorrect answer is one that doesn’t achieve the goal. There could be many correct solutions. Some will be better than others and that will be an interesting and educational discussion to have later. I will first give you a little foreground so any new programmers will understand the relevance of the actual problem and why it is stated the way it is.

Often on programming forums you see heated debates about which programming languages is better, yada yada yada... Perl owns *… M$ sucks... This is all well and good when you have full control over the environment and the tasks at hand. This however, is very rarely the case in a professional environment. Many times you need to build upon or integrate with technologies that others have already put in place. Customer X wants to get a data feed from you in a certain format, you can either accommodate them or not have customer X’s money. Which is the choice you think the guy writing your check will make? Very often these requirements are what makes the problem itself more difficult, but we have to work within the domain we are given. There are many factors involved in choosing which language to use or how you use it, but the goal is to evaluate the options and choose the best one for the task at hand using the resources available to you. If everything in the company is built on .NET and you choose to go with Perl, you damned well better have some facts on ROI to back up that decision.

So with that prologue out of the way, let’s get to the problem at hand…

Problem Solution [View] [Reply] [Top]
Posted by Delta Lt dopel On 2008-03-22 21:58:18

Some time has passed so I can reveal my solution to this problem. Only pseudo code will be included in this post. The actual code will be posted as the implementation part of the challenge in a separate thread. This post will serve as a textual method of solving the proposed problem. Many of the details of how the code will actually work will also not be included in this post. This is meant to be a high level summary of the planning stage of the solution. You must first understand the problem and how to solve it logically before code can be written.

The goal of this exercise is to provide a means of passing data between two single threaded client scripts using only an HTTP POST. We must also do this without significantly affecting the performance of the client script. This problem can be broken down into smaller pieces. I will describe them in terms of the source, the messenger, and the message.

The Source�

Here we have our client script. We know that this script is generating data and delivering it to the user. We also know that that this data is dependent upon information being generated by a similar script on another host. The basic client script algorithm is as follows�

begin loop
process_data() &#65533; These details are not relevant for solving the problem
	build_xml_message() &#65533; I have chosen to use XML to pass the data.  A simple single value message is sufficient to demonstrate the solution.  

Response = htttp_post(xml) &#65533; Does as it says.  It simply sends the message and receives a response.

	If (isValid(Response)){
		Process_xml()
	}

end loop

--We assume that this response will come back in a negligible amount of time. We can make this assumption for two reasons.
1. The target of our HTTP post will be local.
2. The target will handle message queuing and communication with the other host.
-- Notice the immediate response (valid or invalid) implies there will be minimal delays to the client script. Thus we satisfy our performance requirement.

The messenger�

This piece is the meat of the solution. For our client to work as expected, we need a bridge that can both receive HTTP posts and distribute responses from both clients. Now to have a central server of some sort would add latency to our client requests. It would also add a level of complexity in future maintenance and over cost of the implementation. With this in mind, I have chosen to have each client run its own local messenger service. This service will act both as a server for incoming requests and a client to pass messages between hosts. The idea is that when a client posts a message, the messenger will connect to the messenger on the other client and pass the message. The messenger will then respond to the original requestor.

main(){
open_listen_socket()
while(listening){
	spawn_connection_thread() &#65533; when a connection comes in, we issue off a thread as to not block other incoming connections.
}
}
spawn_connection_thread(){
	msg = read_socket()
	if(isfromClient(msg)){
		open_connection_to_messenger()
		write_msg(msg)
	}
	if(isFromMessenger(msg)){
		-- We want to add messages to a queue to ensure both ordering of responses and to ensure no messages are lost.

enqueue(msg)
	}
	
	if(qHasMessages()){
		write_socket_response(dequeue())	
	} else{
		write_socket_response(defaultMessage) &#65533; AKA Send back garbage XML so client can continue processing.
	}
}

The Message�

The messages themselves can remain very simple and once we achieve the passing of data, the content can be changed at will. What is important is that the messages need to contain a structure that is standard as to allow the code to determine the differences between messages. So we end up with three XML message types, request, responses, and garbage.

-- Original request from client script
<?xml version="1.0" encoding="UTF-8"?>
<qtp-request>
	<submit>
	<dField>SOME DATA</ dField >
	</submit>
</qtp-request> 

-- Response generated by the messenger
<?xml version="1.0" encoding="UTF-8"?>
<qtp-response>
	<submit>
	<dField>SOME DATA</ dField >
	</submit>
</qtp-response>

-- Response generated by messenger when there was no data in queue to respond to the request.
<?xml version="1.0" encoding="UTF-8"?>
<qtp-response>
<dField>EMPTY</ dField >
</qtp-response>

The implementation will follow soon. I want to give anyone actually working on this some time to code his or her own solution before spoiling it. Good luck and happy coding : )

Problem Description [View] [Reply] [Top]
Posted by Delta Lt dopel On 2008-03-06 17:32:21

I will try to simplify this process by abstracting the irrelevant details and also to protect the innocent. Remember, this is a real world problem. ;)

Imagine a there is a local desktop application being used by the business in the form of a script. What this script does or how it does it is not that important, but it basically makes some calculations for the user and it is built in VBScript. In the past, these calculations were solely dependent on only one user. Then comes along some savvy well paid exec that heard in some back bar room about the power of divide and conquer and decides he wants to leverage it. So he mandates to the team that in order to improve department’s efficiency, they will distribute some analysis and members will work in pairs. For the sake of simplicity, we can assume that there will never be more than two members of a team because the efficiency gained by distributing the work will be negated by adding additional people. So there is the wrench in our current game plan. Calculations this script will need to make are not only dependent on numbers being derived by another member of the team, they are time critical. User A does not want to wait for user B to email his findings before he can move on, he wants them now and even wants them before user B is finished with them. The script already can do an ongoing calculation, so all that is really needed is for each local script to be able tell the other script what it is doing, as it does it.

Knowing your boss wants this done yesterday; you don’t have the time to re-engineer the script. You can however, enhance the script to make additional function calls, but cannot noticeably disrupt its’ execution or the user will burst in flames. After some research and deliberation you decide that the best way to not disrupt the script is to have it do an HTTP POST containing its data. This is very fast and wont be detected by the user. Luckily for us, the script has already been saving its datasets as XML, so this makes transmitting data from the script seamless. The stumper is how do you get one scripts data into the other?

Remember the script running on each local machine is identical and must continue to be that way.

Also remember that just because the script is built in VBScript, does not mean your solutions have to be. Other than doing an HTTP POST, you are no longer confined to VBScript, but certainly are not barred from using it.

Happy solving!

~dopel

i don't understand [View] [Reply] [Top]
Posted by Ker sefo On 2008-03-17 12:45:10

The story is good and all, but there's only a few lines about the task's description and it's written in such a way that the first time I read it, I didn't even realise it was the challenge's description...

Also, I don't really understand where that comes from... Did you post this so that someone can help you with a project at work or is it genuinely something you invented?

The description is a little obscure for me (and I'm sure, for the others as well)
I'd like to see the technical version with a list of concrete points to achieve (and no story).

Some clarity... [View] [Reply] [Top]
Posted by Delta Lt dopel On 2008-03-18 00:45:48

I am sorry if the question was confusing, as the problem itself is difficult to word. This challenge is a real probelm I encountered. My intention was not to get help on this problem as I have aleady solved it. I will post the solution and the implementation in steps. The solution will probably come this weekend. I have been so swamped at work recently, I havent had the chance.

This challenge is basically a few problems rolled up into one. The main objective is to pass data between two threaded scripts without "seriously" interupting the performance of the scripts.

You can assume that the messages passed back and forth can be processed, by the script in a format of your choosing so long as the message format is a defined. It might be a little more clear if I state the flow of the script.

Basic script algorithm:
begin loop

Process data..
Compiles data into a standard message (You can define the template for the sake of passing sample messages)

does HTTP post
receives HTTP response
if response is valid data from other client, process it
else response is garbage, do nothing

end loop

So as you see, we need "something" that sits between the scripts which can accept http posts and send back responses. This entity needs to do it quickly, whether or not is has a message to send.

I hope this clears up the task slighly. Please feel free to ask more questions if you have them.

On 2008-03-17 12:45:10, sefo wrote
>The story is good and all, but there's only a few lines about the task's description and it's written in such a way that the first time I read it, I didn't even realise it was the challenge's description...
>
>Also, I don't really understand where that comes from... Did you post this so that someone can help you with a project at work or is it genuinely something you invented?
>
>The description is a little obscure for me (and I'm sure, for the others as well)
>I'd like to see the technical version with a list of concrete points to achieve (and no story).

[Back To Programming]

CyberArmy::Forum v0.6
Generated In 0.01607 seconds