I plan on spending the weekdays of next week, working on a new project for Minecraft. The project will involve JavaScript, WebGL, Python, CherryPy, SQLAlchemy / Elixir, Mako and HTML5. The requirements will be on new browsers supporting the standards I’ll be using and provide hopefully as much cross-browser support as possible for the technologies. I believe all major browsers support many of the client side HTML5, JavaScript, and WebGL, so it would just leave browsers <ie9 , Firefox <4, and well people should be using the latest Chrome browsers anyway.

Purpose

To provide a tool to upload a Minecraft save file (multiplayer or single), move around the world and save schematics of buildings. They will then be able to copy the schematics and place the schematics in various places in the same map or upload a new map and copy buildings from old save files.

The reason for this is to allow creating new 1.8 save maps and moving old maps buildings and supplies to the new map without losing hours of work. Also, be able to take advantage of the new features in 1.8 in case a lot of the world has been explored already.

Design

The design will be simple.

1. Users will be able to sign in using Facebook or Google, or OpenID.
2. Users will be able to upload existing save maps.
3. Users will be able to upload existing mcedit or previously downloaded web site schematics.
4. Users might be able to choose existing save maps. (providing space isn’t too much, might be a premium feature.)
5. Users will be able to view their save map using WebGL, JavaScript, and move around the world using their mouse and keyboard.
6. Once they want to copy an area, the controls will change to allow clicking a block and moving the selected block around using the keyboard. The controls will also allow for rotating the view to choose other blocks. The number of blocks will require at least 2 selected blocks one at the top corner and another at the bottom corner. The interface will then allow selecting more blocks to change to air blocks. Once that is complete, the object will be saved to a mcedit schematic or custom schematic format.
7. The user will also be able to place saved mcedit or custom schematic format to existing or new uploaded saved map. The controls for this will involved clicking on an area and choosing from the list of existing saved schematics. The user will be able to preview what the change will look like. Once the user has chosen the area, the software will guess as to the ground and convert air blocks to ground blocks and morph the area to allow for placing the schematic.
8. The user will then be able to download the changed saved file.

Why Do This Instead of Projile?

The Python code will be diverse enough to increase my knowledge of Python and JavaScript. Also will focus my OpenGL knowledge and apply it to WebGL. This will benefit me when I finally get into online game development. I’ll want to have experience and knowledge of how WebGL works, so that it increases my motivation based on my previous working experience. Also will allow me to design the code around my mistakes with this project and with Projile. I don’t get enough chances to work with new JavaScript technology and I want to spend some more time gaining experience and understanding of the new HTML5 JavaScript APIs.

This project, I don’t believe will take more than 2 to 4 weeks. During that time I will have increased my understanding of Python and the technologies I’ll be using in Projile. I will also still be working on Projile during my off time.

Why Do This Instead of Playing Minecraft

When you invest over 60 hours mining, smelting and building a project, then maybe you’ll understand. Now spend over 150 hours. Now try to spend that again while wanting do to something else and most of your time goes away. If you only have 1 hour a day and you already spent over 100 hours, it would require that time and maybe less since you’ll be more experienced and you’ll be back to start again. For some people, they can do that and find enjoyment. Other people just want to play the game and live on with their life. Other people just want to cheat.

Also, it’ll be fun for me to do this. I’m confident I’ll be able to pull it off before 1.8 comes out, unless 1.8 comes out in 2 or 3 weeks from now and I’m still not finished.

Not Starting Until Monday 11th

I want to spend time at least starting the Projile project. I’ve been doing way too many things besides the project. Granted, much of it has been reading over Python books and the Python documentation, but much of it has been procrastinating (ie this post and the previous one). I want to at least get Projile to the point where working on it isn’t a chore and I’ll be able to visualize what I’m working on better. I already have a ton of motivation for this project and since it will use existing code and prior experience, I don’t see it taking even the estimated time. The JavaScript controls probably will take the most time and the converting mcedit schematics to existing maps might require a lot of testing as well. Still, I envision it being crazy fun.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

Tasks I want to have done by the end of today or this weekend.

1. CherryPy server test – Started

   Involves a base controller for making sure that CherryPy is working correctly. Fairly simple to complete, however, I am attempting other tasks at the same time.

2. Installer / Upgrader – Not Started

   The installer must fail gracefully, if possible. Will install the initial tables for running the software.

   1. The installer must make sure that the python version is supported, in case the setup.py install wasn’t called first.
   2. The installer must ensure that dependencies are installed.
   3. The installer will check for existing trac.conf and ask if the user wants it converted.
   4. The installer will check for the database configuration script and attempt to create it checking for permissions before asking the user to create it. If it can’t create it, the installer will halt and the message informing the user to create the database configuration script will be output.
   5. The final parts will involve naming the initial project and setting the authentication details.
3. Plugin Support – Not Started

   Going to involve refactoring or rewriting what is contained in loader.py in the Trac source code. I’ll have to make sure that I attribute correctly any code that I copy. I’m probably going to attempt to rewrite the parts using the modules that exist already and if I have any problems going back to how trac works.

   The goal is to enable controller and theme support in plugins. So what this means is that I want to have some additional coding checking the modules for the presence of certain namespaces and files. The way I want themes to work is that the theme can replace any of the themes in the directory. If the plugin is standalone, then adds the controllers, then the views will simply be called by the plugin to use their own views along with the main theme layout files.

4. HMVC Controller – Not Started

   Well, I believe Trac is currently HMVC now, so nothing is gained in this part. What I’m doing is simply keeping that ability and forcing a default MVC for the core parts. I am going to test core parts as plugins. So the browser will be a plugin containing its own controllers, views and models. I also want to experiment with a blog plugin. I need to first make sure that this is even possible and simple. It has to be simple for plugins like the current Trac.

5. Use Mako for Templating
6. Use SQLAlchemy – Not Started

   Involves simply installing SQLAlchemy, checking for SQLAlchemyin the installer and using SQLAlchemy for the installer / upgrader models.

   There is an extension to SQLAlchemy that enables more of the ORM in SQLAlchemy, Elixir is the name, I believe. All plugins and core code will be transitioned to SQLAlchemy models. I keep wanting to enable plugins to install along with the core installer, but letting WordPress be my guide, I’m not thinking this is a good idea.

7. Authentication – Not Started

   This feature of course must be started initially for the administration to work. The problem is that this feature will involve a lot more than simply working with the web application. I want to have Facebook, Google, Yahoo, etc to the mix and allow bridges and plugins to extend the authentication list. This differs from the official Trac in that Trac only allows web server type authentication, which granted, is quite a lot and could be extended to include the above. There is too much invested in online technologies and custom authentication to be set aside. People understand Facebook and Google authentication, having used it for many of existing sites.

   The goal is to also make it easier for end users to use this project and experiment on whether it matters. However, I’m not going to be able to invest a lot of time extending the authentication to more than custom scheme in the database and implement a basic authorization policy for the software. Once the base is set, extending it in the core and plugins will be possible.

8. Administration – Optional, if time available

   The base administration will include basic project support based on the database scheme. This will include plugin and user management.

9. Navigation and JavaScript API – Optional, if time available

   I want to output JavaScript and navigation dynamically based on an API that will build the list for Navigation and JavaScript / CSS files and output them using a template call. The administration and front end will use both of these to ease development of plugins and extensions.

10. Setup.py – Not Started

    Of course, will enable installing Projile based on the setup tools in Python. I also want to extend the setup.py to include additional options for setting up the web installer and the conf script to ease the install tutorial as much as possible. I want to get the user on the web installer as fast as possible and most likely will require testing apache capabilities (if possible) and providing manual means as well.

    If all the person installing has to do is:

    python setup.py install
python setup.py config wsgi|fastcgi|etc PATH_TO_PUBLIC_HTDOCS

    Then I think I’ll be doing something right. This is relatively simple, while not as simple as perhaps PHP would be able to do. Well, why not? I think I might attempt to include a PHP installer that sets up Projile and if it works well enough, might include one for Trac in the future. The major issue will be permissions, but if it is allowed, then it should be a major boon in easing installation. I’m going to ignore the PHP component for now and focus on Python. Spending more than 15 minutes on the PHP solution probably wouldn’t be worth saving 15 minutes of initial setup of the project.

Future Notes

1. Wiki code

   The wiki is so ingrained into the current Trac that stripping it from the core would be absolutely not something that should be done. It is arguably one of the best features of Trac and the power of the code will have to be converted to this project. I envision the conversion simply moving the code to Python 3.2 where needed and refactoring the code to use Python 3.2 where acceptable.

   Whether or not there will be a specific wiki or a wiki as a plugin, I’ve yet to determine. The code will be there, of course, but whether the wiki can be removed or

2. “But Trac already does this!”

   I know that most of the so called features I’ll be adding are found in Trac now and I could simply code my changes as a plugin and be done with it. If the primary goal was simply to add the experimental features, I would do so as a plugin. To justify my decision, I will say that I’m doing this for educational purposes. Converting Trac to CherryPy, SQLAlchemy and Mako / Genshi, will allow me to be more familiar with those libraries and serves dual purposes of messing around with Trac and learning those libraries.

   Furthermore, hacking apart Trac forces me to go through the entire Trac source code enabling me to familiarize myself with the code base unlike I would have if I simply read through the code. To be honest, I have read through the code already and found parts that I passed over without much thought. Tearing apart and piecing back together does not make for a great initial project, but it does allow me to learn quicker and better understanding of the code I’m looking at.

3. Future blog feature

   I envision this simply being an aggregation of existing blogs or RSS feature. I might make it something more depending on the Page support I want to also add. This will never be a fully complete blogging feature like the one in WordPress.

4. The Goal

   The goal is to get to the point where I can use my new project instead of my blog for tracking features. Also moving to the point where I can copy the existing information over as well. In order do that, I’m thinking I need to track the time spent on each of these features in order to accurately test features.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

Yeah, I took a closer look at Trac and well, oh my god! I’m thinking I’m jumping in the deep end without learning to swim first. That is usually how I roll.

Starting Out: Creating Web Installer

I think I’m going to look at two things. The first is writing a web installer that obsoletes and eventually replace the trac-admin cmd line tool. This will also involve at some point moving the wiki to the database. I suppose the next two or so weeks will be involved in this project. I’m going to refactor also how the htdocs is setup. I want it to be more how a web server should behave, but this will be a learning process as to whether the python modules work that way. Regardless, it should be fun throwing a wrecking ball into that and see what comes out of it.

Most likely the end of my attempting to refactor Trac for another year or two. The experience will be well worth it, even if all of it will most likely be unusable.

I think this will be a good start and will teach me about Python install scripts, entry points, and creating packages and plugins. It will also teach me how the various Apache HTTPd modules work with Python web sites and CherryPy. Even if I decide to stop after this, the next two to four weeks will be well worth learning these topics from an established product.

Depth Of the Web Installer

Starting the web installer should be neat in that it will give me experience with CherryPy and passing execution off to Trac. My goal in particular is attempting to replace the installer (setup.py) script. However, this might prove to be a little bit difficult, depending on permissions. In which case, if it can’t call the setup.py, it will instead work off the existing code, albeit probably a lot slower than what it could be. The installer will also attempt to configure the trac.ini script and create a base project, if one does not exist or attempt to discover existing projects.

The plan is to do this separate from the existing Trac code. Other optional goals will be to convert the Sqlite to MySQL or PostgreSQL or simply keeping it Sqlite. I think this will allow me to jump into the existing Trac code without having to completely rewrite giant portions of the code simply to get something working. Also, being separate, it should allow installing existing Trac installations without many or any modifications. Or at least that is the plan.

Rationale For Web Installer

I read an interesting article about Cluelessness and it had a profound impact on me. The problem I’ve had in the past with Trac and why I often cringed at setting it up was that it took far too long. The process has vastly improved and there are Linux repositories that make installing Trac as easy as typing a single command. The problem with downloading and installing Trac has been solved. The problem with setting up Trac is a little bit more involved.

Given how easily and quickly it is to install WordPress, it is no wonder there are many installations. It would be easier, I think, to convince my boss to use Trac, if I could say without a doubt that it would only take 5 minutes to completely setup and get Trac up and running. This time quote does not include installing python and the apache modules, which also only take less than a minute to get up and running as well.

Well, there is much to say about spending upwards to 4 weeks attempting to create a web installer simply to save at most 2 hours as a novice with setting up Trac. However, it is more, I think, about the perception, which is to my boss, that it isn’t worth the time investment. Also, with existing and awesome online bug and project trackers, it is a even easier to simply buy an account rather than spend any time setting up a private tracker.

While this task is, I think, a little much to be cutting my teeth with starting to really truly learn Python and Python web development. I think the complexity now is something I’ll rather enjoy rather than starting at some point after I’ve started. In other words, I’ll be damned if I’m going to attempt something like this after a couple of months and I’ve learned as much as possible about Python and realize just how crazy attempting this actually is. Right now, ignorance is bliss and I’m thinking, “How hard can it be,” and deep down, I know it is impossible, but I’m still a bit unsure. I’m going to attempt it and when I fail, I’ll be all the wiser for the attempt.

Lessons

1. HTTP modules WSGI, FastCGI, SCGI and standalone (CherryPy).
2. CherryPy web site development
3. Python setuptools module
4. Permissions revolving around calling setuptools in a web installer.
5. Permissions and also hopefully downloading and installing optional plugins in a web installer.
6. Setting up Trac using a web installer.
7. Creating a simple, easy to use web installer providing some initial configurations.
8. Optional SQL Alchemy for converting Trac databases.

Task List

1. Successfully get CherryPy accepting HTTP requests with the aforementioned Apache modules.
2. Test whether there is an existing Trac project setup and successfully bypass the installer (preventing the installer from even being started) to start Trac.
3. Attempt to call setuptools in the web installer and figure out if it is even possible even with permissions.
4. Attempt to create a Trac Project trac.ini file.
5. Start replacing and complementing the trac-admin command line script.
6. Unit tests.
7. Release for cash and profit

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

I’ve set up the environment, in a virtual machine with Ubuntu 11.04 with Apache HTTPd modules for mod_python, SCGI, WCGI for python3, and fastcgi. I’ve set up the domain and I plan on downloading my Trac GIT repository. I think part of the problem is that I’m going to have to get intimate with GIT command line (which is similar to SVN, so not much of a problem there, might take a few days to get comfortable and then a few months to become expert enough to not need a guide.

I’m switching to CherryPy as the HTTP framework. It natively supports all of the above testing environments, so I won’t need to work on adding support to Pyramid. I think CherryPy is also somewhat inline to what I’m learning for in learning Python since it appears to be doing it in the way of Python.

I still haven’t started coding, part of the problem was setting up the environment and the other parts has been reading up on Python programming. I don’t think I should just jump right in, because the code will definitely won’t be in the way of Python.

I do think that Sunday, I will start by actually running Trac, updating my current Trac fork of the GIT Trac repository or create my own from the SVN. I want to at least start on the HTTP framework part and see how far I can get. I’ll have to review the code points and ensure that I capture everything from the current Trac. Once that is done, I’ll probably commit, push and go back to reading about Python programming.

I wish there was more of a Zen of Python Programming that tells you exactly how you should correctly program Python, but I think at least my fumbling around and looking at CherryPy will be beneficial to learning The Way.

I’m not exactly sure about the Templating, I’m sure I want to do Genshi or support it, but I might be better off learning Mako. I still have quite a ways before I need to make a decision. Also, I’ll probably want to look over the database schema for Trac and look at models in SQLAlchemy. Probably be refactoring Trac and adding multiple project support in the process. I’m interested in plugin support, but I really think that enough of it has to be built-in for anyone to actually use it. Probably have a base amount of information and client and company support to get the multiple project support up and running.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

I know I wrote a few posts a few months back on creating a new bug tracker with a few ideas derived from other products and from online essays. I think that instead, I’m just going to take Trac, an existing bug tracker or project management and work on it instead. This gives quite a few benefits, while saving a lot of time and motivation.

The problem is that there are many bug trackers or project management web applications for PHP, Ruby and Python. Creating another, well, my motivation was pretty much soaked up in frustration of starting yet another project that would go no where. I also liked Redmine, but I think they have their head in the game and I’m excited to see where they take their web application.

Working on Trac helps me learn Python, learn how Python plugins work, learn how an established web application handles Unicode, Views, etc with regardless to Python and web application development. Unfortunately, my coding probably is not going to be on par with the standards of Trac and so, well, I’m not even going to attempt to make it to where my code can be made upstream. I have faith that anything I can do, the developers working on Trac can do better.

So I created a GIT repository forked from the official GIT repository (http://github.com/jacobsantos/trac). I haven’t started working yet, but I plan on implementing a few ideas of my own. I hope no one worries about this forking Trac, but I’m unsure how far I’m going to get and if I’ll actually succeed in completing the two or three months projected. I’m just doing this for shits and giggles, while learning on the way.

Using Pyramid For Controller

I guess one problem is that Pyramid, from the docs only supports WSGI. Trac on the other hand supports mod_python, FastCGI and CGI. I want to support mod_python and FastCGI and not exactly lose the existing support. For that I guess I’ll need to modify Pyramid to add support for those two. At the beginning, I’m going to simply ignore the lack of support.

I plan on refactoring the controller implementation in Trac and moving it over to Pyramid. I think it would be fun to see how Trac’s controller works and get Pyramid working as a replacement. I think that any Python web development I do will be using Pyramid, so I’ll like to take an existing web application and move it over.

Using SQLAlchemy For Models

I don’t know how Trac does the models, but refactoring Trac to use SQLAlchemy should be a fun project. What I do know, I think makes working on Trac a bit more difficult. I would like to think that SQLAlchemy would be a nice addition, but whatever, for my research purposes, I hope that it will allow me to quickly prototype certain features and get the messy parts of Python MySQL development out of the way.

Web Based Installer / Upgrader

Meh, one of the things I hate about Trac, err, well, at least for version 0.10-0.11. I haven’t actually used the recent version, however, with the refactoring, I’ll be sure to add this. I thinking it will be similar to the way WordPress does this. In that the only thing the configuration file has are debugging and database details. My hope is that if everything is configured correctly that it can serve most needs.

Web Based Accounts and Administration

Yes, Trac already has these, but what it does not have is, I think, Facebook, Google, Yahoo and OpenID integration. I like the way GetSatisfaction handles this and Google and Facebook user integration has made other web sites signing in a breeze. I wish, in my effort to make Trac more user friendly, to support as many integration as possible. As well as provide an authentication bridge, if Trac doesn’t have one already. If it does, I’ll probably just hook in or refactor it to support these ideas.

1. Registering – Creating an account either for the Trac install or from an existing integration account. All tickets will keep track based on the internal account system, which I believe is the correct way to integrate the external services.
2. Permissions – Trac already has this. It will be simply a matter of tying everything together in with this system.
3. Plugin Management – I believe Trac already has this, but I want to extend it with a few ideas.
4. Reports / Reporting – Part of my research includes reports (graphs) and other features I plan on getting into when the time comes.

Refactoring Plugin Support

Since I’m moving to Pyramid and SQLAlchemy, I plan on experimenting with enabling and disabling plugins. Allowing plugins to register their own routes with security preventing overriding the existing routes. The plugins will also be able to create their own tables and configuration.

I think this will be fun. Trac already does a lot of plugin support and I want to pour over how Trac does this and experiment with changing how this is done. I don’t know if any of this is possible, but I think it’ll be fun to find out.

Making Wiki a Plugin

I never liked the Trac Wiki and well, I think making it a plugin will allow me to move my focus from it to other parts of the system. I do however like the ability to add pages and content and I wish to add pages as a plugin as well. This might allow Trac to be more of a CMS, but I don’t really think people use the Wiki except as a portal to the ticket and project management.

I don’t plan on stripping everything of the Wiki out. I like the external wiki support and the parser for comments and pages and plugins will stay. Meh, with everything else, this will a low priority or not at all task. I do still plan on adding RSS feeds support and pages support to either replace or supplement the Wiki.

Focusing On Ticket

One aspect of GetSatisfaction that I like is that creating a new ticket is on the front page. I would like to move the focus on tickets to the front, instead of the often irrelevant information on the front page. I also like that GetSatisfaction simplifies and doesn’t force you to sign in until after you are finished, saving your information until you have signed in. I don’t know if this is the best way to do this, but I plan on allowing for both.

I need to look at Lighthouse and see how it is doing its ticket creation. GetSatisfaction is more for end users and Lighthouse for programmers, so there are going to be differences between the two. I think allowing for both and letting the admin decide, if there are differences would be better.

Multiple Projects

GetSatisfaction works by clients and I think that is fine. I do wish to implement this independent of developers and of clients. Projects will exist outside of clients and companies. The problem and focus I think with this is well, I’ll be adding a lot of functionality that won’t be applicable to many Trac installs. I think the best way to implement this is by hooks and allowing for plugins to implement the required functionality.

I want to experiment also with reducing the functionality until it is used. I think that if companies are not used, that it must not be part of the URL for a project. I think that if a project is not created, that it must not be part of the URL. That is to say that there will be a default of everything for when companies and multiple projects are created. It is just to hide it until it is used.

The difference between clients and companies is that a client is a single entity and probably will be no more than a single name in a dropdown. Companies are an organization entity used to allow projects to have duplicate across companies, but unique within companies. Sort of how GitHub works now. I have jacobsantos/trac and Edgewall has edgewall/trac.

Reports

I like how Ohloh has graphs and reports on projects and I wish to duplicate that for Trac. I’m not going to do the same graphs and reports, but they should have a similar look and feel. I want to list developers and contributors and have graphs for their contributions to the project. I want you to be able to click on a commenter name and see what they’ve been doing.

WordPress uses a Props counter and I think I can reproduce that for users as well. Create a sort of profile with an API to access what users have been doing. I do believe that majority of what I’ll be doing once the foundation is completed will be here and improving this.

Agile Development

Part of my research is creating a system where Tickets are assigned to developers or contributors to allow them to focus on tasks that are front and forward. This will include a time estimating and tracking system broken up in chunks. What this means is that if a contributor says they can only devote 8 hours a week and they place on their plate an estimated 10 hours of work, they will receive a notice warning them that they might be taking on too much. It will not bar them from accepting more, but simply note this.

There will be several ways to record actual time. I think interfacing with existing time tracking cloud based solutions will be best as well as providing an API for retrieving and setting time for tickets. One that I like is Toggl and it has an API that I’ll work with. There are other cloud based solutions that provide an API, which I’ll implement as well. I also want to provide a custom solution, but that will come much later. Providing support for existing cloud based solutions should work well enough for now.

So keeping in with the graphing. I hope to display the estimated verses the actual for projects and tasks. If tasks continuously take longer than the estimated time then that will be tracked and displayed for contributors and programmers. There are other fun graphs that I want to experiment with, but for now, these are the basics, if I can implement these, then I think I’ll have a great start.

Weighted Bug Assignments

I read an article or essay a while back that gave an equation for determining the weight for a ticket based on severity, priority, estimated time, etc. It was a weight for how important or likely the ticket was going to be worked on based on multiple factors. I want to play with this, but I think it will be fun to use on a hidden level at least. The research the person did in the blog post was to go on and say that by utilizing the weights and time better, the programmers were able to knock out more bugs and move unimportant bugs to the next release.

Statistics

I want to implement as many statistics and reports as possible based on these statistics and graphs based on the reports. I believe it will be fun to do as well as spice up the product. With everything I’ll be implementing, I don’t think I’ll get to the graphing parts until towards the end. It will provide an incentive to continue development for research sakes.

I think also, I want to sort of provide an opt-in for passing the statistics on to my server, so that I can go over and analyze the data. I do know that many will not do this and I do think that not many people will even use my version of Trac to even pull this off, but I hope that at least a few developers do… for science.

Support Tickets

Why the hell not?

Ripping Off Ideas From Other Sources

I hope to rip off ideas from Redmine, Lighthouse, GetSatisfaction and other bug trackers and project management sources. If there is time. I don’t plan on spending more than 3 months on this project and if I get everything above done, then I think I’ve had succeeded. Really, for my personal projects, I hope to at least use my version more and if I see something from other source that I like, I plan on stealing that idea. I might need to write down where I got all of my ideas for legal reasons, but I don’t think it will be that big of a deal or at least until it is.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

Merging with GIT is quite simply the most beautiful action I’ve every attempted. Even more so after the SNAFUs that have been caused with Subversion. This not to say there is anything wrong with Subversion. I think with all of the debates of GIT vs Subversion or why use GIT over Subversion kinds of misses the point of SCM. This is a topic for another blog post and one I’ll unlikely ever write. Debate somewhere else, certainly, but meh, I might write it at some point.

How SNAFUs Are Born

User error. That begs the question of the best way to use Subversion to minimize the impact of SNAFUs. I think if all you are ever going to do is commit to trunk without ever branching or tagging, then you’ll do fine. The reverting and history tools are great. However, in business and large projects, problems are borne from branching and not using the best practices of Subversion. This is however a topic for another time, perhaps in Part 2, where I will go over possible branching methods.

What Are Best Practices?

I really don’t know all of them. I only know the ones I remember from my readings and from personal experience and from the work place. Someone more experienced and a veteran of SCM will have more and better ones. For the sake of my personal explanations to why you don’t do something a certain way, I’m just going to explain it this way. If I’m not doing a good job and you have improvements, then please say so. I would rather use them for my personal benefit for teaching newcomers to Subversion.

Really, given the work place fast pace environment, it is difficult to say, “Create a new branch for it.” When it comes to Subversion. I know that some would say that the cost isn’t that high, but when the repository is 1GB download, branching becomes a serious PITA. Also, if it is easier and quicker to not create a new branch, even when one is required, then a new branch is not going to be created. These really are best practices for branching and when branching isn’t used as much as it should.

1. Always Merge Revisions That Haven’t Been Merged Before Yours

   Damn, I can’t explain how many times this simple advice was not taken and how many SNAFUs it has created with Subversion. I tell people this again and again and they don’t get it and wonder why everything has gone to hell.

   …Except that this is not a perfect world and people do not always create a branch for stuff that could not go in immediately. This means the responsibility of the person committing a simple fix in a branch has to take up with original person who committed the revision that can not be merged for one reason or another. This is partly why branches should be created, but knowing when a new branch should be created is not always easy. If this is the reason, then certainly talk to the person and get to testing and working around the problem.

   One action that should not occur is just going ahead and merging and committing the revision anyway. One caveat to this is that if the prior revision that wasn’t merged does not touch any of the files that the revision being merged modified, then it can safely be merged.

2. Commit Additions, Deletions, and Modifications Separately

   If you do, then you’ll never have problems with having to go back and commit revision dependencies.

   Given that this advice is not going to be taken seriously, it is a sure bet that the realization of why you should do this will become very clear when the first SNAFU comes into play. I will also say that for a while I haven’t done this and it has bitten me in the ass enough times that I’m starting to take this advice myself.

   A better way to put this is never commit an addition with modification and the reason will be clear after a few revisions when you only want to commit a modification or addition. The problem is that merging a revision merges everything in that revision^[1]. The problem with modifications is that usually the reason it wasn’t merged in the first place is that it wasn’t ready to be merged and those modification revisions accommodate. If you have additions to one of those modifications that are not ready and need to merge those additions to the trunk, then in order to complete the merge, you’ll need to also go and merge all of the revisions that haven’t been merged that also had modifications for files in that revision and also in the revisions that you’ll need to merge.

   If you always keep additions separate, you can merge those without fear of merging something that should not be merged and keeping track of all of the revisions that the modifications touch.

3. Rule of Thumb for Branching

   If it will take more than one day, create a branch. If you are unsure whether or not the modification will be accepted, create a branch. If what you are doing is highly experimental and might break anything, create a branch. If you are refactoring major parts of the code and know that only you will be working on it and will go live after you are finished, create a damn branch! In fact, unless you are making a minor change that is going to be merged immediately and doesn’t belong in an already created branch, then just create another branch. Just for shits and giggles, if you have the time and it won’t screw anything up, create another branch.

   The problem is that too many times, at my work at least, the expectation was that the change would go in immediately, but plans changed and it could not. All of a sudden you have a ticking SNAFU bomb just waiting to go off, unless it is in its own branch that no one else is working on. Even now, there are revisions that can’t be merged for one reason or another and I cringe every time I see those revisions. Luckily, they affect only a small area and the impact is very small, so much so that you could just think of that area as its own branch since no one enters that area.

4. Commit Often

   Well, I don’t have the full explanation, but it is very good advice for those that use it. The gist of the advice is that if you commit often then reverting or going back to a prior change won’t having a huge impact to the other parts of the code. The other part of it is that you’ll want the history of committing new features. Having many small commits helps with merging and minimizing conflicts when merging.

   In my personal experience, given the overhead of committing using TortoiseSVN, in the 3 to 4 years of using Subversion, it has only bitten me once. Really, I understand in part the rationale for doing this and I do follow this advice sometimes when I’m using GIT, but until the cost for committing in Subversion becomes cheaper, I don’t think I’ll be doing this.

   I added this because well, any best practices wouldn’t be complete without it.

Footnotes

1. This is not completely true, it is possible to merge a single file in an revision. However, for the purposes of simplification, it is better to always merge a full revision. I do know that in 1.7 of Subversion, the need to do this is minimized. However, the amount of problems with merge-info and reintegration and merging is painful enough to just always do it this way.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

I’ve only been programming Java for a week, but I’ve learned quite a bit in that time. Well, mostly from making a lot of mistakes. Generally, most mistakes are made during the learning process with a new language. Trying to build a damn GUI Application with a multithread proxy server was one of those mistakes.

Let me explain. I’m using Swing or more accurately a Swing Framework that comes with Netbeans, but not the Netbeans platform. One thing I’ve discovered through trial and error and then doing some researching, is that Swing only allows you one thread. Well, this isn’t completely accurately, but for all purposes, if you attempt to create a Thread and attempt to access any Swing object, your application is going to crash and burn. Also, at any point, your thread might run and deadlock the GUI thread causing your GUI application to become unresponsive.

So yeah, if you want to use Threads with Swing or really any GUI framework, except I think you might be able to get away with it if you are using OpenGL, but it is tough building a working GUI using OpenGL. I’ve came to the conclusion that I was doing it wrong half way through the entire process. The reason I didn’t scrap my current work was that I still didn’t have the proxy working correctly and I didn’t want to have to completely rewrite the entire application. I didn’t think I had the motivation to rewrite everything again and not even have a working product.

Well, I have both a working product and a GUI application. The solution was to use SwingWorker for the threads which basically adds the Runnable to Swing to run when it pleases and won’t lock the application. The problem remains is that I still need to create a thread to run two tasks side by side. A better description would be to say that I need to thread something that I don’t really care about when it finishes and appears to take forever to complete. It is currently slowing down the Proxy and there isn’t much I can do about it, unless I wanted to create another SwingWorker, but the problem is that I want to queue the messages sent to the thread and handle other tasks.

I want to create a Timer which will shut down the thread and close the connections if nothing had been received during that time. Currently there are stability issues with the GUI application not accepting any connections after the Server drops the connection or the application is closed.

What I thinking about doing is creating a new thread whenever a client socket is received, which will start a timer thread. There is currently a thread for the client to the server and another thread for the server to the client. What I’ll end up adding is another thread which will manage the messages sent to it and also eventually handle an OpenGL window application provided that it is allowed.

I’m thinking about keeping the GUI application. I can finish work on the separate process for the Proxy server and have the GUI manage the options and fork the Proxy server from the application. It would work almost exactly as it does now, but just work better and faster.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

I’m almost finished with the hard part of the HTTP library and the Hook library is almost complete as well. The next two projects are currently in the planning stage to be started once those two libraries are complete enough to be used in projects. I want to develop a ORM / Active Record Database Library to understand the workings of other database ORM and Active Record libraries. I also want to work on a Fuzzy Logic library for PHP game development.

Fuzzy Logic Library

I think that if there was already a Fuzzy Logic library for PHP, I probably wouldn’t even attempt an implementation. Given the math involved, I’ve been putting this off for quite a while. It has been almost 4 years since I wanted to do this. I believe I’ve progressed far enough with my programming skills and also understanding of what is involved with Fuzzy Logic to attempt it.

I don’t believe the initial attempt will be usable. I also think that I’m missing some key parts for understanding what exactly is part of the implementation of Fuzzy Logic systems. Reading what little I’ve found, there appears to be math that the authors assume that I should already know.

I’m going to start development of the class structures based on how I want the implementation to work. I believe that much I do understand, even if I am missing details. If I’m still missing information and the unit tests are not working, then I’ll have to buy several more books in order to complete my education. If I have to do that, then I’ll be charging for using the library to earn back some of the money I spent. If I don’t, then I’ll probably buy one or two of the books just be sure I didn’t miss out on features that people expect, but the library will be free for all to use.

I’m not exactly sure, I might be able to complete certain math related implementation details not dealing with set theory. I think that is part of it really. The graphing equations are really just geometry and I’ll simply need to look up the formulas for those and implement it in the library.

ORM / Active Record Database Library

I don’t like how other database libraries implement ORM and I want to development my own to see if how I think it should be implemented is even possible or feasible enough to be implemented. I want to keep the library simple, but feature complete enough that I could use it in my own projects. I believe that if I were to do a public project, that I should use another database library.

I do want to spend the time to learn how ORM is implemented and develop my own database library. I also hope that this time, the library I write actually works. The other attempts either were shit or too convoluted to even try to complete.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development

I’ve been developing a HTTP library for PHP on and off for the past 3 months. The motivations is to create something that should have been written for WordPress from the beginning. I think the HTTP functionality is okay for what it is supposed to be, but there are design limitations that this new library seeks to address.

For one, the ability to process headers without having to hard code the functionality into the library. Also, it removes all dependencies from WordPress. Really, I believe some of the hacks in the current HTTP API in WordPress should be dealt with in a better way. There will still be possible to do many of the things WordPress has with the hacks.

I’m also rewriting quite a bit of header processing, not only because some of the current code I don’t hold copyright to, but also to take advantage of PHP 5.3 and to fully conform to the RFCs.

You can find the code at Git Hub.

Possibly Related Posts:

• Game Engine Development and Open Source
• Plans for Base CMS
• Project Plans
• Calibre Improvements Part 2
• Usability Testing Software Development