Automated Refactoring: Expanding and Collapsing

I want to do several experiments with automated refactoring or really refactoring based on a limited set of rules. The experiments are based on WordPress and I want to write out my ideas. I find it helps to write out an idea to see better understand it. It might amount to nothing, but that is the point of experiments.


Expanding is taking function calls within a library, framework or application and sticking replacing the call with the code in that function.

Collapsing is reducing code in a function or method to smaller functions or methods.


The point of expanding is to remove other refactoring that may have been done to functions already. To see the bigger picture so to speak to see whether a better pattern is witnessed. Sometimes, you need to see how code structured or restructured to see whether you could improve it.

The purpose of collapsing is to break larger functions into smaller functions that handle a single part of the larger function and used in the place of the code. The goal is to see whether the smaller functions can be reused in more places. To test a hypothesis that smaller functions could be reused in more places than larger functions. If the functions are used in one place or have limited usefulness, to slot the smaller functions body back into the functions that call it.

There is a larger goal with collapsing classes and taking the methods and reducing them to methods and executing the same process over these methods turned to functions. Refactoring the objects to either arrays or data classes with public properties. This process seems like it would be more involved and I will attempt it after the expansion and collapsing steps are completed. The collapsing process will be difficult enough to accomplish without rewriting classes.

Finally, the final part of this will be to generate tests for the collapsed functions. Really, the point of the entire exercise is to generate functions and eventually classes that can have unit tests generated. Generate the test objects and integration tests and integration test objects.

Test Cases

The expanding and collapsing process is meant to produce code that is unit testable. Then also generate the test cases, the test objects and expand on it to integration and system test cases. If this goal could be reached, then producing test cases would be a walk in the part. It should hopefully reduce the amount of test cases that have to be manually written.

The goal is to produce code that is of higher quality from existing code bases. Creating a tool that could be developed further to work on other projects.

The difficulty is telling the type of a variable and what data to place. Part of this will be manual, meaning most of the test cases, test objects, etc are generated with only the data and validation algorithm being needed to be completed. The point being that the person would go down each section and fill in the missing parts until all of the test cases run.

Secondary Separations

For WordPress, the goal is to separate filters and actions from the functions and methods that call the plugin api directly and move them to its own function. There isn't much point to this, other than to see at a glance how many filters and actions exist and where the calls are located.

The HTML and JavaScript will be split from functions and files that have it now. It will be assumed that all PHP files are doing it wrong and create a separate file. Any code will be split from that file and placed in a function.

I also want to move functions to namespaces and reference the new functions from the original.

I want to introduce inversion of control, where each function has an interface and default implementation that could be replaced. This will hopefully help with testing and creating tests. The smaller functions will hopefully have functional paradigm principles applied, where the input and output are idempotent.


Every function and class is reformatted. I still have to complete the output to match the WordPress coding standard. The other reformatting is for inline documentation. I'm going to apply some standardization to inline documentation. Eventually, this will apply to JavaScript and HTML, but for now, I'm more concerned with completing the PHP parts. 

The HTML and JavaScript will be handled by separate processes that will work on the HTML and JavaScript. Most likely not written in PHP. The JavaScript will probably be handled with a Node CLI script. I haven't decided how I'm going to handle the HTML.

Keeping Progress

I think I'm going to introduce a JSON or YAML file to handle the manual modifications that are needed and replaced when the script is run. Especially when namespaces start being applied.

Machine Learning

I haven't decided how I'm going to do this. I might create a simple web service that takes the expanded functions and asks users to refactor the function and expanded classes. Saving the results and learning from the human refactoring. I want to see how well I can get the machine to refactor and write tests for existing code.

For now, I think I'm going to stick with the manual process. It is good enough and I have a lot more control and can tweak the results a bit more, for now, than I could the machine learning results.

WordPress Foundation Does Not Want Its Dev Community

The conclusion after being confused for so long is that the WordPress Foundation does not want contributors to core to determine the direction of where WordPress is heading. It took a while, but I think there is enough evidence to suggest this is quite the possibility. I would love to hear what the response would be.

I predict it would be something along the lines of, "We'd love to hear feedback from trusted developers who contribute to core." Except, the fact is more that what they should say is, "We'd love to hear feedback from trusted developers who contribute to core that already match our decision."

The decision to stop supporting wp-hackers mailing list by Jen Mylo could be seen as trying to remove noise from signal and provide a better mechanism for discussions. From the reality it seems that noise included any and all criticisms of what Automattic and the WordPress Foundation was doing. Included in the noise, actual discussions of better solutions or solutions of where WordPress should go.

Before I left the WordPress core dev community in 2009, it seems very much that under Jen's direction that external discussion of where WordPress should head was very much not appreciated. The response was always, "Wait until the planning meeting for next version and we'll discuss it then." For which my response was, "I have to wait 2 to 4 months, before getting feedback or discussing tickets that could go into this release?"

Damn! That demotivating as all fuck. I tried to join the WordPress core community again in 2010 and the response was the same. The melt down came when Jen decided to post something towards how the community could work towards bettering WordPress or something similar. I was incredulous towards that post given what I experienced in the community over the 18 months I was working within it.

My Experience Was Esoteric

For the longest time, I just figured that since I was an asshole, that I got special treatment and was pushed out, because fuck me right? I just contributed a little inline documentation or some shit. Right? I suppose I'm very much butt hurt from that discussion. When Nacin decided to threaten to blackmail me, I'm said, "Fuck this! If the core team is going to steep so low as to threaten to blackmail someone, then this is not a community I want to be part of."

You had a toxic individual telling me I was toxic and wasn't welcome. Fine. I wasn't being paid to work on WordPress. I did that because it was fun and I was learning. After it stopped being fun in 2009 and in 2010, I just decided to drop WordPress and do other things with my time.

I thought my experience was an edge case, but I would be wrong. WordPress doesn't like individuals that take it upon themselves to improve WordPress in ways that are not inline with what WordPress Foundation members have decided.

The WordPress core team hides behind this idea that they are the protectors of the User and need to stick to the 80/20 rule. That users are what matters and therefore anyone not jumping on their direction need to go away.

Is 200 Really a High Number?

My question has always been, "If WordPress allowed more patches and gave more feedback for patches, then would they have more contributors?" I never knew because I never had a frame of reference. WordPress is proud that 200-300 people contributed to WordPress.

I would question whether they would have more, had they not shunned programmers attempting to improve WordPress. I would question if WordPress would have more if they adopted Git model of version control and allowed others to improve WordPress and test WordPress independent of the WordPress Foundation.

I never knew what the answers of these would be. Even with the high number, given the amount of people that use WordPress, you would think that number would be higher. I wouldn't consider the contributors high, until you get to 1000.

Also, the last time I ran the statistics, the report was that majority of contributors submit 1 to 3 patches for 1 to 2 tickets. The statistics is from 2010 and I'm planning on refreshing the statistics in the future with proper reporting. I'm curious what the number of major contributors would be since there are companies that pay employees to work on WordPress full time. I would guess the number would be around the same.

My guess would be that the more people with commit access going through and looking at bugs, the more contributors. That does seem to hold out, but adding a mere hundred more contributors still seems small.

Do As We Say

It seems to me that the WordPress core team doesn't care about what you think, only that you contribute and fix the bugs that are in the Trac. You might say that this makes sense right? Fix bugs and contribute to core. The problem is that WordPress itself is a bug.

What I mean by that is that the way WordPress is developed is prone to errors and is extremely difficult to accurately and easily test. It is more than just refactoring for refactoring sake. It is more, refactoring to break WordPress into chunks that could be isolated and tested separate from the overall system.

It does not seem members of the core team are interested in doing this or at least allowing someone to do this outside of their plans.

This has always been an issue of WordPress. WordPress Foundation and its members do not seem to understand Quality Assurance. They paw at QA wondering if they can consume it or if it is poisonous, but they have no idea what that term means. Or at least give the impression of it.

The reason they don't care about it, is that it isn't a concern they have at the moment. They are content with bumbling through WordPress development wondering why everything is burning down around them. If they stopped to wonder about getting a team together to actually prevent the fires in the first place.

Then well, you would have to have a competent team managing WordPress in the first place now wouldn't you? The first conclusion was that the core team is incompetent and wouldn't know good code if it kicked them in the teeth. I don't believe that conclusion is accurate. Many on the core team have many years of experience, so it is difficult to believe that would be the truth.

More than likely there are those who wish they could improve WordPress. Really improve WordPress, but dare not, because the key players would strike it down like they would any external contributor.


WordPress Foundation doesn't give a fuck what you as a developer want to do to improve WordPress. They want you to just shut your face, put your head down and fix bugs the core team themselves created. You be a good boy, oh yes you are, and maybe, just maybe, they will allow you to join their club.

Just don't create any waves, because they will drown you.

The Perfect Programming Language

There are a few thought experiments I've been running through my head for the past few months. The questions that lack importance, given that they've already been answered. I can't help but ask anyway.

Why do languages have their own VM?

For most languages, you are limited to their custom engine. Certainly, programmers have made Ruby, Python and PHP for various virtual machines developed by Oracle and Microsoft. These are popular languages and for popular virtual machines.

The question is more broad. If I want to run Language X, then I need to find interpreter or compiler for my system. If such a thing exists, some of the more interesting languages have poor support for Windows.

My thoughts go beyond simply running code.

Why can't I use library written in Language X in Language Y?

A few things has peaked my interest in this thought experiment of mine. Technically, you should be able to do so using LLVM, given that if you compile to IR for a given language, then you should be able to call into that module from another supported language. There are details that are no simple and the first is that it would require having a reference to that library to know what the call depending on the restrictions within the caller language.

The second is HippyVM, which attempts to provide a VM for PHP using PyPy. One of the long term goals after successfully porting PHP to PyPy is interoperability between the two languages. Where you can call a PHP library from Python and a Python library from PHP. This could change things. Or maybe not.

The problem is the lack of success in Java VM, CLR and LLVM. It is certainly true in the technical sense that you should be able to call another library in another language, because it is after all, bytecode. The truth is that it is not easy and if something is not easy, then it is not often done. My criteria is calling into userland libraries and not built-in types and libraries, which are obviously supported. 

 Best practices, debugging and programming tools.

In the thought experiment, You have a VM that can run all languages that exist and will exist. In this VM, there is interop between languages and libraries.

So what do you get?

  1. Debugging tools and libraries that work in one language can be applied to other languages. In this environment, you don't require duplicate effort of porting libraries from one language to another.
  2. Programming tools from one language could be applied to another. So if you have a tool that formats code based on the VM AST, then could that be applied to another language as well? So formatting code to a coding standard would be as simple as running a command.
  3. Learning best practices from one language and applying it to another. If there was only one VM to code to, then what tools and best practices might come from it?

I think about all of the neat tools, libraries and code concepts that are coming from Node.JS, Ruby, Python, Clojure, etc. I wonder where it will go from here. Where would it go if every tool had a foundation for building off of? If it wasn't difficult to convert the JavaScript package tool to include every other language that is part of the VM.

If the tool to query the running debug state of JavaScript was just inherit of the VM and I could do it for PHP or JavaScript or Python.

DSLs and programming should be easy.

In my thought experiment, I think about the different concepts I've been learning from functional languages. How awesome these concepts are, and if they could be applied to my favorite languages.

I keep thinking back on the Python language and how everything should be as readable as possible. How everything should make sense as much as possible. You should be able to code how you think it should work and most of the time, it should just work. That is perhaps the best feeling in the world.

Yet, there is also a fascination with making code as small as possible using only symbols for most things. I weep when I think about how great functional languages are, but how complicated they are when starting out. There is a great power with Haskell and Clojure that I love and would enjoy.

It is true that with any language, there is a learning curve and I don't believe or at least there is no evidence that it will go away. BASIC and COBOL prove that even with the most simple of all languages, you still have a learning curve.

My point, is that code should be readable while not completely going into the realm of BASIC and COBOL. I'm not sure how to accomplish this and I think looking at Python and other languages that have this goal in mind should give some insights to a possible solution, if one exists.

Haskell does give one possible solution that I want to explore with experiments, if at all possible. Domain Specific Languages, might allow a programmer to provide an intuitive interface to a library as long as building the DSL is not complicated. If creating a mini language was simple, then would more programmers create them to provide an easier way of using their libraries?

It might create a problem where you have too many tiny DSL and programmers have a difficult time remembering and learning them all. It is one possibility and I'm not sure what the solution or if it is even a problem.

Future of Telecommunications

The future of telecommunications will be a backbone that instantly transfers data across the world and beyond by using quantum entanglement. The experiments already show some promise, but given the results, it might be another decade or two before this is fully realized.

I don't believe that the miles of fiber crossing the oceans will be removed and replaced with the new technology immediately. The Internet is built on reliability and given the failures that exist now, it would be while before trust is applied to the technology.

Source note: I wish I could provide resources, but much of the papers require membership and the Wikipedia, from the last time I checked, hasn't been updated. Given that the post is speculative and future experiments and research will further the available knowledge on the upcoming technology. It would be like writing about the limitations and possibilities of quantum computing. Each year would bring new information that could contradict and prove incorrect what is currently provided in this post.

One of the current problems with quantum entanglement is that it is currently being manipulated by lasers and therefore such a backbone will require being housed in a building. Similar to the mainframes of old. Some new technology for detecting and changing the polarity of the state. Recent articles of possible states put the current limit at 5 states. This should allow for sending a nibble with a single atom, except that the paper stated that each state had to be checked separately. Whether, this is a limitation of the technology, I believe the article also concluded that advancements might provide a means for reading all states at the same time.

Given that entanglement can be applied to small crystals, I have wondered in a science fiction fashion that perhaps graphene could be entangled with a correlated pair in a sort of solid disk state. If so, then the mechanism to read and write would be smaller and allow for more bits to be transferred. Even if each block is limited to a single read or writable state, then laying and, or stacking the entangled areas should provide enough area that GiB or TiB of data could be transferred instantaneously between entangled pairs.

I believe that early devices probably will be warehouses transferring much smaller packets. Also, given the cost and patents and other factors, the price tag of such a venture would be limited to research facilities. Much like the early networks and Internet.

Given the current technology, it appears that it won't be seen in homes with near instant access to the Internets. I predict that the 'arcane' way of changing and reading the entangled state will be improved; shrinking in size until a device could be carried in the pocket.

Probably at a time when I'm an old man, telling the younguns, "I remember a time when we used radio and Gb bandwidth was a luxury. Now you young folks can download hologram movies faster than you can play them. Gee wiz, it is a good time to be slaves of the robot overlords!"

The anal probing isn't so bad once you get used to it.

Sources and further reading:

Mageborn Series

It is awesome that independent authors are publishing really entertaining stories. It gives me hope and ideas on what styles work best.

The first two books of the Mageborn series has been on my Kindle App for about 6 months. I bought it on a whim given Amazon's recommendations and the cheap price. I haven't had time to go over all of the books that I bought. Last week I decided to finally get to the books.

Mageborn has been quite entertaining. What I mean is that it flows extremely well. It reminds me of Brandon Sanderson where the story doesn't burden you with too many descriptive details. It still goes into the magical systems within the world describing how they work and their functions.

The story starts off like it would be an extended D&D campaign but quickly moves on to a more of Harry Potter adventure.

It is difficult to truly describe the style of the author. He has a little of Scott Lynch, Jim Butcher, Brandon Sanderson, J K Rowling, and Orson Scott Card. In this, he has be defined as more unique style.

The stories have the heart of Orson Scott Card, the adventure and simplicity J K Rowling and the story weavings and adventure of Brandon Sanderson. The story doesn't hold back punches and is dark like Scott Lynch while maintaining the mystery and somewhat quick pacing of Jim Butcher.

I am extremely happy I finally decided to read the stories and look forward to more from the author.

As an aside, it kind of reminds me of Servant of a Dark God. Well, there are too many differences. I guess it only truly reminds me of that because of the crushing oppression and tension of the enemies and the fact that magic has been systemically been weeded out and almost destroyed. 

Fantasy Programming and Women

I was thinking about women in fantasy and science fiction. I heard that the reason J.K Rowling didn't use her full name was that she was told that people wouldn't take the books seriously if they knew she was a women.

That didn't make any sense. Some of the best fantasy writers have been women. Male writers have a tendency to write descriptors about landscape and tangential nonsense. Reading Books of Magic opened my eyes. Certainly it is a young teen book series. It was written beautifully focusing on the characters and the plot.

Moving on to how this applies to IT and programming is perspective. Creating solutions and paradigms is a matter of perspective. Having multiple perspectives on a problem has a greater chance of picking at a better overall solution.

I have the belief that writer's build upon the styles and success of those that came before. If that holds up and I believe that the current movements to code standards and quality future perspectives should give insights to creating better solutions from paradigms that work.

Not only that, but I think also that for those perspectives to work they have to have some success.

What do I know? I think it is silly that there is even a conversation and there has to be an movement to get more women in programming. If the question is, "Would women be successful in programming field?" then the answer is WTF? are you talking about, do they have a brain? Yes. There is your answer.

If the question is, "How do we get more women into IT and programming?" Well. As with anything, a person has to like the field and think that it is fun.

Truthfully, I can't answer that. Eventually, it will get to the point where it is no longer a question and simply be reality.

Game Engine Development and Open Source

I need to remember to do occasional write ups on the development of the game engine for the continuation of the PHP game series that I started years ago. However, I'm stuck relearning OpenGL for OpenGL ES 2.0.

Decision: Open Source

I was debating this internally for a while. Well, the game was always going to be open source, but the decision was whether to charge for the initial product. I've decided that the market for the game engine is small, so either I would have to charge a lot and spend a lot on marketing and advertising, or charge a little and hope that the market is a bit bigger than I thought.

My motivation came down to support and lack of market. I believe there are quite a few people who are interested in developing a WebGL game engine web application. I'm kind of hoping that those people gravitate to my project instead of Three.js or another complete product game engine. There are already paid game engines based off of Canvas and WebGL. I'm kind of hoping that if I can get a working product out before the end of the year that I can make something that others probably have already started to develop.

Well, the point is, is that the more people whom are able to support the product and do support the product, the less I have to support. I'm also hoping that smarter people join the project, so that I don't have to spend months researching and developing prototypes and then ultimately developing something that sucks.


  • I'm lazy and hope others help out with the project, thus giving me more motivation to commit to the project.
  • Less support I'll ultimately have to provide for fixing bugs and extending the project with new features outside the scope of my games in development.
  • More people using the project providing marketing and name recognition.
  • Mainly this is an educational project and when you start charging, it stops progressing for learning and starts progressing for money. I hate that type of worry.


By the end of the year, I plan on meeting or exceeding these goals.

  • 1000 active players a month (shoot for weekly) for Farmer Sim (tentative name).
  • $500 a month in revenue from Farmer Sim game.
  • 3 active developers working on the base game engine. Myself included. I know a few people that might be interested in a working project, but not at this stage.

I think that if I can have the graphics component finished by this Summer, then I should be able to have a playable version of the game by the Fall or Winter. It is my believe that given the scope and playable features I might have in the game that quite a few people will be interested in playing the game. Experience has shown that at least 100 people will sign up for a new game without entirely even knowing what the game is about or if it sounds cool.

Farmer Sim, I believe is an interesting concept that I believe should have a lot of players. Given the almost infinite land scope, it will beat out anything Flash that matches the genre. If implemented correctly and with the features, it is possible to be bigger than anything current on the market. There is a slight problem. The game is not meant to have a lot of players. It will not scale well without a crap-ton of servers. Which is not to say that it is impossible. It is to say that given the start-up status of the game development, I will be unable to afford the hardware that would be able to support 100k active players.

A bit of a problem certainly, however, I think that if I do sell the source for Farmer Sim (or give it away), it is possible to extend the landmass by allowing multiple server instances to combine and create a wider complete landscape. However, the payment system and logistics for that would be difficult, not the mention the implementation security and cheating problems.

I still do not know the full extent of the storage problem. It might be less than what I fear and in which case the total amount of players could be higher. Also, as the game brings in more money, more and better servers will be possible, extending the capabilities further. Not that the processing power is of significant concern, mainly storage at this point. Storage will always be a problem, given the implementation of the landscape.

However, I'm not concerned with having a 100k active player base. I want no more than 5k active player base. I feel that it is hard to interact with the players and I really do want to interact with a small player base. Also, 100k is unreasonable, the game won't ever reach that point. My goal should be to allow for 5k players or 10k and limit to 5k just so that everyone is happy and enough land and resources are available for everyone.

The problem will be capturing 500 players with as what might be only a month or two at best case and a few weeks at worse case to reach this goal. It is possible to have a working version in 3 months, if I worked on it every day as if it were a full time job. The problem so far is that I don't know enough of the graphics component to really haul ass. By the time I get off of work, my mental capacity is spent and it is really difficult to comprehend what I need to comprehend to make it work. Not impossible, just frustrating.

Once the graphics component is moving along, the game graphics will progress quickly and I might just be able to work on it as if it were a full time job (devoting 5 to 6 hours on the weekday). I'm going to set this month to finish the graphics component and have experimental storage and API support. If I can do that, then I'll have another 3 months to decide between Python and PHP and figure out the umpteen other features. oAuth and OpenID seems to be a hassle to develop, except maybe not as hard as what I think. I believe that once I start working on OpenID and OAuth, I'll find it is fairly easy and should have a library working in 3 to 4 weeks. Which gives 2 to 3 months working on game features and administration.

My other goal is statistics. I want to sort of be data mining players for the purpose of finding what works, what doesn't work. I like statistics and sort of get into UI and play testing. I would love to develop reports on how many players have multiple accounts and how many people are signing up daily, weekly, monthly. How players develop their farms, cities, strategies that players develop. Exploits that players find. It is all very fascinating. It'll also be interesting to try out different payment models and see which one works out.

I don't want to make a lot of money, and I doubt I will anyway. I simply want to see what the difference is between subscription vs in-game cash vs free game and advertising. It'll also be interesting to see how people respond to other types. Such as a sort of donation model, where once a given number of players have paid a given amount of money for that month, everyone receives premium features. It really depends. I do hope to make some money given the amount of time I'll be spending and have a little something something left over. It is simply that I don't want to give myself hope of something that may never happen.

Plans for Base CMS

I've decided on using Zend Framework version 2. It isn't yet released, but that shouldn't be a problem. It also has many of the components I was planning on writing myself in such a way that I was planning on coding. This saves a great deal of time and heartache as I feel the framework is above and beyond my requirements. My main goal at this point is simply to start programming. If I can overcome that part, then I think January should be a good month to have a lot of work done.

The main part of the base application is to have an administration panel, probably modeled in some part on WordPress administration, a user management, and the main web site. There will be a module based system, but the goal isn't to have a true CMS where modules would extend the mostly empty application. The goal is to have a base application which can be extended into other applications. The modules would then add additional functionality based around the 'theme' of the base application.

I plan on creating two applications extending the base application: Directory and Gaming Engine. I plan on developing this projects concurrently as much as possible. Well, I plan on doing modular coding regardless, so my guess is that if I do this correctly, I should be able to take one piece meant for one application and use it in another application.

The goal of the gaming engine is to build something similar to Minecraft. The voxel engine of Minecraft proves that it is a viable visual 3D platform. The games will not be clones of Minecraft. Actually, one game will be very similar to Sim Farm, Sim City, Stronghold and other such games. I loath to go into more detail until I work out the base game engine and have the visual components working. The final product may end up completely different depending on what I eventually find that I can achieve in a relatively short amount of time.

Regardless, one game will not have any real threat and is primarily market and simulation based. I've been wanting to create this game and Minecraft has given me the inspiration on how I can build the visual parts. It has been frustrating in the past, since I didn't know how to work the game out visual within my ability or lack thereof.

The other game, Mecha Asylum, will provided I am able to create the voxel game engine, be what I intended for the original game in 2004. Similar to the farming simulation, there will be threats and will have enemies.

Having both games very similar in design and a voxel engine to display 3D designs, the development should go fairly quickly. I also want to have a way to create 3D models in game, but most of the ones I create will probably be by code.

The directory will be completely designed around social networks: Google+ and Facebook. Google+ is constantly adding features and the same applies to Facebook. The goal is to provide a portal and then simply use the social networks to store certain data. Authentication and Authorization will be based on Google and, or Facebook sign in. I will eventually to have create a custom authentication based on user and password, but none of the current projects require it. The amount of people who don't have a Google or Facebook account are fairly small and the numbers are constantly raising. I will eventually add Openid and other third-party authentication, but I figure the big two will work just fine, since well most sites are implementing them too.

I'm going to develop the Directory without ad support in the first version. With the theme support with base application, it shouldn't be that difficult to add a third-party ad system to the application. Also, whatever solution I create probably won't be as full-featured as Google, Microsoft or another ad site. Regardless, I think one of the first implementations will be to actually support some of the more popular ad sites.

However, I'm getting ahead of myself. This week, I wish to finish a basic administration panel, an working authentication sign in process using Google and Facebook. A basic profile that pulls from either Google+ and, or Facebook with additional content. That is it. If I can get this working, then I should have motivation to continue next week and start working full time on both the game engine and directory applications, while moving some code back into the base application.

That reminds me, as I develop the game engine code base, I plan on blogging about the process. Some of my most popular posts are the game development posts and it would be interesting to continue the series that I dropped so long ago.

Bullet: E-Book Library Management and Content Server

I have a solution to the problem and Windows 8 gave me some inspiration. I was thinking about writing a Windows 8 app that implemented some of the features of Calibre, but thought it would be a lot of work with little pay off. I want to work on Windows 8 apps, but I don't think right now is a good time. That doesn't mean that the solution couldn't be web based and it also means that the Windows 8 solution will be mostly complete. The goal shouldn't be to replace Calibre. I know that is foolish. Calibre is awesome at what it does and has plenty of time and effort going into it. What I want is a easier library management, searching, and content server. I'll be stripping out the majority of the features of Calibre and focusing on simply the library element and the content server. Therefore it really won't be Calibre, it will be a separate product that clones the content server for devices that support it.

It'll also depend on Calibre, since I'm not going to be implementing the e-book conversion. Calibre has a command line conversion tool that I will use. Thus using the best feature of Calibre and not spending months developing my own conversion tool.

I'm working on the throwaway first version and have the directory structure. I'll be using for the web controller, Mako for the views, and SQLAlchemy for the Model library. Given the awesomeness of Django, I might switch to that at some point. I simply want to prototype a version and see how well it goes.

I found out one of the use cases for my dislike of Calibre. The problem is that it often copies files for its library for organization. The issue I think is that the original file still exists, creating duplicates. The reason, I was told by an independent party, that if you had a temporary location: removable or network drive, then it is better to move the file to a location known by Calibre. The issue is that it also does this with hard disk drives that won't be removed at any point in the near future.

It seems then, that I have to check whether the drive I'm on is a hard disk or removable. If it is even possible to know. I think I'm going to ask, until I research a way to figure out how to tell whether a drive is removable that is also cross-platform. I think this must be the reason for its decision. It simply seems inefficient and could create a circular problem when the Calibre library is in the directory you are pulling the books.

Actually, I think I have a solution, but it isn't something I'm going to do in the first version. On Windows at least there are various options and one of them is to access the Removable Drive option list and provide an option for loading the e-books on the removable drive to another location. The first version will have an option in the settings to allow for transferring a location to the content library location.

I think this compromise would be neat. I can't wait to get started on the e-book library management. I'll probably be spending the next couple of days working on this and see how far I get. It isn't however the two projects I wanted to work on. However, the work on this will help with the project I'm having the most difficulty motivating myself.

The name "Bullet" is a play on Calibre and is tentative until I find a better name or end up liking the name and keeping it.

Using ZendFramework 2 beta1 For Directory Project

I was already thinking about using ZendFramework 1.11, but since ZF2 is at the beta stage, I think I'll just jump on that wagon since my project isn't going to be ready for another few months anyway. ZendFramework2 also has all of the primary base features I'm looking for. I'll probably have to hack some implementation details for missing features that will be released in a few months as the version matures. Still, I should be able to hack together a prototype and hopefully be able to implement many of the features I have planned during my vacation break. Well, I'm unsure how far I'll get, if I do have something, it is unlikely I'll be able to release it since, the project will unfortunately be unstable until ZF2 is released. I figure I'll probably release something after 3 months anyway. The primary pieces I'll be using, the Controller, plugin and modules are fairly stable now (I believe), so it is possible to release something.

I'm also guessing that the authentication interfaces and adapters are fairly okay to implement or extend, so it is possible I'll be able to implement authentication. Something I was planning on doing anyway for shits and giggles. I'm unsure how far developed the database component is, but I'll probably build my own anyway in case I want to move to CodeIgniter or Fuel later.

Besides that, I hope to start Python. I figure if I work 8 hours on the directory project a day and then 4 to 8 hours on the Python project. I should have worked a pretty productive vacation. Also the projects should be mature enough to warrant further development for the next couple of months. I'll have to pick a project to continue after this week, since it is unwise to focus my attention on more than one project at a time.

I had wanted to complete the HTTP library project and at least have something complete. However, I lost motivation on that once I start thinking more about the Directory and Trac project. I have far too many projects on my mind. I was kind of hoping to complete the HTTP library project and get that stable and off of my mind. However, I'm hoping to show the Directory project and Python project some love this week and see how well I progress.

Most likely I'll go back to the HTTP library project one or two days and finish up majority of the foundation coding and have something that could be released before the end of this month. Well that is the plan at least. Even if I keep scattering my plans based on what I'm feel like coding this moment.

At some point I really want to do something with Calibre, but I'm unsure what. Probably nothing for the next couple of months. I want to do some D programming, but these PHP and Python projects keep raising to the top of my thoughts. If I can remove them or get to them to the point where other people can take over, then I think I'll be happy.

Project Plans

Finish HTTP Library

I've been putting this off since the last weekend for catching up on TV and movies, plus working late. Given my plans for my week vacation, I need to have the majority of the code completed by Friday and off course, I haven't worked on it this weekend either. Given that this week is the deadline for the current project for work. I'll probably have another few nights of working late or arriving early and leaving normally. I prefer the latter. I might take Friday off, given the extra time I've been putting in to the project.

Regardless, I'm thinking that this week should give me enough time to at least estimate how much more time is needed to complete the project. If it looks like I can complete the project over next weekend, I'll attempt to do it. I hope to at least put two hours a day into it this week and attempt up to 5 hours a night if I don't work late. I believe if I were to do that, the project should be finished this week. Or at least I'll know how much longer it will take.


I want to start the Projile project and devote the 8 hours I would be working at work on the project. The rest of the day, if the HTTP project is not finished, could be devoted to the HTTP library. I think given that I'll both be learning Python, doing research, and testing as I develop that the time will allow me to have the motivation to start and hope get to a point where I'll continue.

Also I'm back to using Trac as a foundation, but I decided I don't have to refactor everything and I can work on the controller and model parts separately from the core Trac until I'm ready to convert the rest of the application. Well, basically, it is the same as if I'm creating a new project management application from the ground-up, but it will be less demotivating since I'll be using Trac as a base.

I'm not sure how far I'll be able to get given that I'll be attempting something new with Python development. I've mostly written scripts for simple tasks and nothing this big. I'll probably work on what I wanted to work on last month, which is the installer and plugin model. If I can do both of those, then I'll be really excited to continue.

Oh, I'm unsure how far I'll get during the vacation time. Probably not too far, but I plan on putting the project on hold after the vacation or maybe working on it for another week or two depending on how much motivation I have.

Also, both the "Alloy of Law" Mistborn Novel and another book is coming out the 8th of November. I'll probably spend a day or two reading those as well.

Fuzzy Logic Library

I've been wanting to write this library for a long, long time. Almost 3 or 4 years now. I think I'm finally ready to start the throwaway prototype version. It should be near what I've been wanting. The goal of which is to make it easy to use and prevent a lot of setup and have some database adapters as well.

I'll probably attempt Test-Driven Development for the prototype to work out the API and math. The difficult part is really just defuzzifying the results. I had wanted to include debugging, which is simply displaying a graph of how the Fuzzy Logic data and results would look like. I'm unsure if I'll be able to get to that with my time I'm going to allocate to the project.

I'm allocating 2 weeks to the prototype. I think that will give me enough time to see if the API works with ease of use and if it is usable for defuzzifying the results. I'll skip inline documentation or at least the phpdoc kind and simply work on the unit tests and code. I figure I'll be throwing away most of the code anyway, so there isn't a point writing documentation. Also, if I can get the code running, I can see how well it performs with fuzzy logic results.


HTTP Library - October 31 to November 5-6. Projile - November 6-7 to November 20. Reading Fantasy Books - November 8 to November 12-14 (Probably 5 to 6 hours a day. Alloy of Law probably will be finished the 8th and will take the entire day.) Fuzzy Logic Library - November 21 to December 5. Projile - December 5 to May 2012.

Other Projects

Quickbooks Connect Library

I'll probably hack this shit together within 3 to 4 weeks and see who wants to test it. I have plans to port the PHP to Python. Actually, it is completely possible this could take 2 weeks.

PHP Database ORM Library

Basically take the CodeIgniter ActiveRecord Library and the DataMapper library extension to CodeIgniter and package it in a standalone library. None of the code from CodeIgniter or DataMapper will be used, but it should be mostly API compatible. Again, probably hack a version together in 2 or 3 weeks, writing tests matching the API of CI and DataMapper, then fill in implementation details. If it functions well, I'll probably simply package it as version 1 after writing inline documentation and user guide.

Search Engine

Port Nutch to D and, or Python and interface with both Sphinx Search and Lucene Solr. Really, simply just a spider that implements a dumb version and a Nutch clone. Shouldn't take very long and might be something I attempt during my vacation time. Well, the dumb spider and interface with Sphinx and Solr at least; the Nutch port would take considerable more amount of time.

PHP and Python Game / AI Library

Something I'll probably attempt during the second half of 2012 before I start working on the browser game engine. Might also implement several easier algorithms or at least prototype different Game and AI APIs. Might also take a few breaks from Projile during 2012 to work on Neural Networks and Genetic Algorithms. Graph APIs, Pathfinding and random terrain generation might also be fun.

I'm guessing that some open source browser game engines will exist by 2012 as well. It is possible IE 10 will support WebGL as well. It should be interesting to develop a HTML5 JavaScript library that doesn't require backwards compatible code and simply implements workarounds for missing HTML5 technologies. Several of which already exist anyway. I can't wait to start developing something that I've been wanting to do since 2003 and 2004.

The Way of Kings and Cosmere Theory

I've been thinking about this too often and I think it would be better to just write my ideas and move on. As always, I don't believe my opinion is valid, until the author decides, well nothing is right or wrong. It is possible that the author might even incorporate a really good idea into their story in their own way. Not that you would know, unless the author states that to be the case. Regardless, I have been thinking about Cosmere Universe a bit. Major Spoilers, read at your own risk

Theory About Adonalsium And Cosmere History

I'm unsure where I heard it, I believe it came from one of the books Brandon Sanderson wrote or might have been from The Way of Kings, if any of his books at all. The idea is with sufficiently advanced technology, how would you know it isn't magic? If you were to send back a TV to medieval times, they would probably think you were a witch and that it worked on Magic.

The idea of Adonalsium is similar to me. From the text of The Way of Kings, what if Adonalsium was a device of technology created by beings and it went horribly wrong? The Shards were men at one time and therefore it holds to some reason that they were mortal and came across the power somehow.

There are other theories, of course. Given Japanese mythology found in games and other science fiction, it is possible that the explorers came across Adonalsium in their journeys and never created it. Experimenting with either the device or substance caused the shattering of the device or substance into shards which were absorbed into the men and women around the Adonalsium.

I do quite frankly hope Brandon doesn't state the beginnings as Earth. It is great when it was the first time and if he does end the mystery that way, then hopefully it is the first time someone has came across it. I'm not saying it is terrible to do that, The Sigilverse did it that way and it wasn't terrible. I did groan, but meh, I also think that any explanation will be disappointing. It is nice to discuss the possibilities.

My point is simply that given the current understanding that Adonalsium could be technology far more advanced than anything in the current worlds. Well, the same could be said about the Cosmere. If you discovered that you could travel through what requires "magical" means, then how else would you explain it? That is to say that it could simply be another name or explanation for "Hyperspace".

Why My Theory Is Slightly Flawed

Brandon Sanderson doesn't follow normal rules of science fiction and fantasy. His worlds combine elements of both actually. Not completely fantasy in the traditional sense and not completely science fiction either. The problem is that the Gods really are Gods and the magic really is magic. It would make sense then that technology wouldn't be in the traditional sense either.

What is obvious is that there use a single world where the Adonalsium existed, whole and unbroken. Events long before any of the current books, something happened that both fragmented Adonalsium and made "gods" of men. I'm interested in knowing the history, because it seems quite intriguing. Given what Brandon has been capable of in his previous books, I expect something I hadn't even thought of. I suppose it is a limitation of my mental comprehension and lack of creativity.


There are hints in the book. Damnation must be an physical place, because after all, it is possible to bring the dead back to life. It is after all extremely difficult to do and seems to require the power of Gods to do it. The fact that it is possible proves that the dead do exist on another plane accessible by Gods and in theory by some forms of magical systems.

It is in my opinion possible that the easiest way to travel the Shardsmere is death. It is possible that everyone ends up at the same place in death and it is a terrible, terrible place. However, given the similarities of Damnation to Hell, it is probable that the atheist or anti-Damnation might exist. There has been so far, no proof of this. Most likely Damnation is all fire and brimstone because of Rayse taking it over and making it so. If Damnation exists on the Spiritual plane of the Cosmere, then it makes sense that Gods would have control over how it functions and looks.

Gods have limitations. Rayse can't simply thread over the other Gods and can't go against his shards nature. I find it difficult however to believe that the Spiritual realm would be limited in dimension.

Not a lot of information to go off of. I kind of like the stories where people exist waiting to be born, kind of like the old mythology tales of Greek Gods. Again, not a lot of information on the Spiritual realm, so far.

Why I Enjoy The Cosmere

I enjoyed the Sigilverse a lot when it came out. I couldn't help but to think, "The only thing better than this would be if it was in book form! That way you could extend the universe in ways you couldn't in comic form." To me The Cosmere is that idea come to fruition. The Branson Sanderson books in the Cosmere isn't exactly what I was hoping for, since I kind of wishing for Sigilverse stories by many authors, kind of like Star Wars extended universe.

The main reason I think is the foundation that Brandon Sanderson is setting up for future authors. It is possible that having a multiverse could get old very fast, depending on how many shitty authors try to saturate the genre. The only other problem is that there are just so many ideas for multiverses, or at least from what I can predict. I think that with every multiverse that is created and extended, it is possible that the ideas will become better. You can see the evolution of Fantasy, Science Fiction, Zombies, Vampires, etc where the modern interpretation makes the classics seem dull.

The only problem I have really, is that Brandon has set up his multiverse with many books and many of them aren't even written and let alone published. It will probably see him make a lot of money, but I do hope that he lives a long, long time. I also hope I live a long time as well. The build up and anticipation so far has me wondering, "What will the final history of Cosmere be?"

Why I'm Most Likely Wrong

The problem is that I assume that each planet is in its own universe, creating a multiverse (not the correct term for it, but whatever). Even if each planet is part of the same universe or even same galaxy I don't think it matters. Given the distances and improbability of interstellar travel, the distance between the planets doesn't matter. It is simply a matter that the shardsmere be the only want to travel since normal technology would never allow you to traverse the distance if multiple universes or even multiple galaxies. Well, this does assume normal physics rules and I know that doesn't apply in science fiction.

One of the themes that I think Brandon uses is that the mind allows for interesting possibilities. Meaning that it is possible only with the mind what will never be possible with technology. Using a short story. It might be assumed that Shardsmere is the only way to traverse large distances or even multiple universes.

Native Inhabitants

I'm starting to go off on tangents, but I like to get as much thoughts out as possible since many of them are wrong or at least not well thought through. At the very least, I hope to get them out of my mind so I can think about other subjects. Also, I'll like to put my own thoughts and interpretations out there before I go off and read other theories from people.

Regardless, I've assumed that like in Mistborn, that the humans were created. I'm starting to think this is incorrect, except maybe in Mistborn planet, where the gods did in fact create the people on the planet in the likeness they seen before.

"In Time" Movie Premise Flawed

Disclaimer: Obviously, I haven't seen the movie and basing this post off of the trailer, which is ironic given the accusation. I will revise this post once I watch the movie Friday. I just want to get my initial thoughts out to the one or two people who might accidentally come across this. As always, this post is for myself and anyone is welcome to comment or join in.

What I Believe The Theme Is

I believe the creators of the movie believe the movie is about Capitalism and the inherit evils within Capitalism. Given the recent unrelated events on Wall Street and around the World and USA, this movie might hit the right nerve with people. The problem, I think is that the allegory use of time instead of money is so exaggerated that the economic system they truly appear to represent in the movie is actually communism. The irony is that they created something similar to Animal Farm (according to what is in the Trailer), that I don't think the message they are attempting to get across works.

It is without a doubt that people are going to read into the movie what they will and the point that people might believe it is hitting against Capitalism or Communism, depending on what they perceive. Or people might just enjoy it for what it is, a Sci-Fi movie with a really awesome concept where the plot device mostly takes a backseat, hopefully.

What I Believe the Premise Is

The premise is that Capitalism is evil with its creating of the two tier working class verses upper class society. Capitalism keeps the working class poor, while filling the pockets of the rich. From the trailer, it appears that the bosses keep the working class down by institutionalizing increasingly higher and higher quotas for which the workers are basically slaves. The working class is shown, what little is shown, working their asses off, while contrasting to the upper class who appear to barely work at all.

Why The Premise Is Flawed

The premise is flawed based on the economic fantasy that is created in the movie. If the writers are in fact stating that the system is Capitalism, instead of what it actually appears to be, Communism, then they are using the wrong system or at least creating a satire of Capitalism so extreme that it might as well be Communism. If I am correct, then I predict that the movie won't even touch upon the alternatives to how a person join the "High Society" and given that most workers die before they are 30, aren't even given a chance even embrace the alternatives.

It wouldn't be an interesting movie anyway. Who wants to watch a movie where a person is repressed, develops an awesome idea, markets that idea and then makes a ton of capital from that idea? Not me, it would be boring as shit. Actually, watching a movie about a guy taking a shit would probably be more interesting than that movie. Regardless, still they would allude to the idea by taking briefly about some guy who actually achieved it. That would be Capitalism.

Why The Movie Is Actually Communism

I'm not talking about the unrealistic impossible communism where everyone shares everything and everything is peace love and free sex. I'm talking about the real world communism where you have two classes, sounds similar to the movie right? I'm talking about the Soviet Russia, China, Cuba, North Korea, etc. I'm going to ignore China for the most part since they've since adopted many Free Market ideas since the past decade or two and are doing quite well because of it.

The problem with Communism is for the most part it is run by people. The people who run the government do not want to be subject to the same "equality" of their citizens, which means you end up with a two tier system. See Soviet Russia, Cuba, North Korea, etc, with exception of China in the recent years. The problem also is that there is no graduation system from the class. If everyone is equally poor, because the finite wealth that exists after the higher class takes their share is spread equally. Is it really better since no one can move up?

It is fair system. Everyone exists at the same level of poverty or if the rulers were actually competent and the levels were great for everyone, it might put everyone at some level of middle class. Which ironically is where most people in the US are at anyway. What it takes away is the incentive to work hard to move up. If all you had to do was work at the same pace as everyone else with no reward to work hard, then why should you? A few will work hard, simply because that is their nature, but I doubt 1) it will last long and 2) other people would enjoy being shown up.

From the trailer, it does not appear that workers are able to move up from their class. From the "cop", it is assumed that if you didn't have a lot of time and then get a lot of time, that you obviously stole it from someone who did. The fact that time (money) is regulated in such a way to prevent one class from raising above their class appears support the claim of Communism.

Why Movies Get Capitalism Wrong

This is not the first movie to bash Capitalism in a incorrect way. I remember a number of years ago a movie where the main character is frozen and wakes up 500 years in the future to where he is the smartest people alive. That movie blames Capitalism and its inherit greed with creating more and more solutions which dumb down consumers and create products people don't need to gain more and more money. Also has the side effect of inflating the monetary system to where a dollar is virtually worthless. Well, I could be wrong. It was either that or Governments general stupidity with lobbies and inherit corruption.

The beauty of Capitalism, is that if you have a good idea, put in the time and effort to prototype it. You either save up the money to start a company or use the prototype to gain capital from investors, whom in turn fund your company to one day get a return on their investment. Anyone can do this or at least anyone who has a good enough idea, can either save up the money or convince enough people to invest and run the company in such a way to be in the black after a few years and start growing year after year.

How "In Time" Might End

The reason why I like the movie is that the main character is a reluctant hero, which by events out of his control turns him into a modern Robin Hood. You are supposed to believe that the main character not only should give away time for free, but that time should be available to everyone regardless of how hard they work. You are supposed to feel revulsion against the "High Society" for being lazy bums who milk the working class like slaves. You are supposed to think that the cop is the bad guy and the guy who dies in the beginning is a martyr worthy of a freakin' statue or at least a religion.

My thoughts, the cop is going to die, horribly. The main male character is going to have crazy amounts of sex with the female lead, which will be totally hot and not at all rape because she is totally willing even if technically the male lead kidnapped her and is keeping her against her will. The main male character will either go on to distribute all of the time to all of the people and not at all find that the equal distribute means that everyone dies at 35, but probably around 70 or 80 or institute a change in the system where people are more free and capable of living longer, which is practically the same thing.

By having the "evil" cop quote the supporters of Capitalism, it will prime the minds of the audience that actual logic supporting Capitalism is "evil" because that "evil" cop said the same thing. will add the "evil" cop to their list of "Villians Who Were Actually Right".

How I Want "In Time" To End

I want Justin's character to be horribly raped while the female lead watches horrified and then watches as the female lead is shot in the face and he is shot in the gut to die a slow horrible death while thinking about how bad he failed.

Or at least shot in the face while almost succeeding at the end. Then the woman feigns ignorance and goes back to her life pondering what might have been. Then like several years later, shit hits the fan and the working class raises up and is slaughtered. They finally realize that he who has the big guns usually wins hands down and if you think your life was bad before, just wait until you attempt to raise up and find out how bad it really can be.

Completing HTTP Library For PHP

I'm going to spend the next couple of weeks completing or finalizing the HTTP Library for PHP. The reason for the delay has been that I was unsure that the direction I was heading was the correct one. After realizing that yes, it was the wrong direction, I had to figure out a better path to take the library. It has been many months, but I believe I finally have a good implementation for completing the library. It also seems that I'll be able to refactor a lot of my existing code and hopefully release a working version within a few weeks. Well, refactoring might not be the correct term since I'm scraping a lot of the classes, creating new classes and consolidating a lot of code into 3 main classes while creating new "Feature" classes for other parts. The Transports will probably also be completely rewritten for the "Feature" classes functionality.

The key is the ability in PHP to access protected class members, even if the instance is not the same. I found this by accident while creating an ORM Database class and wanted to access the table name and other properties without making the members public. After testing that the code does in fact work as it should, I abused the hell out of it to great affect. I'm unsure whether it is a bug in PHP and if it will continue to be available.

I'm guessing that the behavior is intentional since it is "correct" in the technical sense. I'm unsure if purists would be accepting of the behavior as it does appear to violate encapsulation in the pedantic sense. Technically, there shouldn't be anything wrong with the behavior since the members are protected and the child classes inheriting the parent are in fact interacting with the protected members. I'll post more information once I complete the library and have a set of code available.


Right now, I'm current going through and cleaning up the code and created a base class for the request and response classes to inherit. The feature classes will also inherit the class, which hopefully will allow for some neat magic functionality to happen behind the scenes without having to do some tomfoolery. It will actually be nice, since I'll be reducing the classes to 4 main classes, instead of multiple subclasses. The problem was that the old version had to pass the body and headers, along with the request classes to get the plugins to work correctly.

If my theory works, The plugins will be simple and reduce the esoteric code in some of the classes into their own contained classes. My goal actually is to strip the code down to the bare functionality and get the transports running tests. Once that is finished, go back and start implementing the features. In a sense, the transports will still be independent and the request class is a wrapper for the sending the correct data to the transports.

Inline Documentation

I'm adding inline documentation to places where it is missing and whenever I add a class member, I'm adding inline documentation. I'm hoping that with version 0.4, I can simply focus on getting a stable version and work on unit tests and additional "Feature" classes. I'm thinking that version 0.4 will be the stable beta version, which I'm going to market to early adopters. Once I get enough people using the product or I'm using the product for several months without any issues, I'll release version 1.0.

User Guide

I'm going for the gold standard with this and the other libraries. Meaning, I'm not releasing the final version until a proper documentation, which includes an user guide for developers on how to use the library. I don't believe the user guide will be all that large, since the functionality is going to be reduced.

Unit Tests

I'm going to have to modify the existing unit tests to work with the refactored code. I'm also going to have to finish the tests that are missing for classes. Like the Inline Documentation, I don't believe this will be all that difficult and I'm up to writing the new tests. I know where I'm heading and most of the code and tests are done that it isn't as daunting a task as it was months ago. I look forward to releasing a product that is professional with a level of unit tests of what a professional product might have.

Well, it is a library, so certain level of testing won't be possible, but I'm kind of hoping that people look at the quality of the documentation and testing as a factor for using the library. I'm unsure if this will be a factor towards someone using the library or not. However, I'm going to market this as one of the features.


I'll give it two weeks. Tomorrow I hope to have more of the Message, Request and Response code completed pulled from Settings, Header and Body classes. After that, I'll have to go back and modify the unit tests and run them. If I have time tomorow, I'll look into modifying the transports. If not, then really majority of the remaining time this week will be attempting to get the transports working. Once the Transports are working, I'm going to focus back on refactoring and moving the remaining code to "Feature" classes.

If the code isn't completed in two weeks, then I'll probably set it aside and continue reading for a bit. If it looks like another week would finish the project (library code works and unit tests are done, but only requires user guide), then I think I might go ahead and spend the extra week. I just don't want to get caught up with the project for a month or two when I have other projects to work on.

Calibre Improvements Part 2

The last post went over a lot of improvements for the GUI side, but really, my main concern is with the library backend system. This is most likely the component I'll work on first, since well, it should be done within a reasonable amount of time and also should have an immediate improvement. Calibre appears to use a directory structure for managing the library, along with a database. I haven't yet looked at the code, but it appears to copy the file. I'm assuming that a symbolic link is created on *nix systems. However, I'm assuming that it is a copy on Windows. Regardless, I don't believe it should be even using a directory structure in the first place and should use the original directory for any conversions and reading.

I'm sure there is a reasonable explanation for doing this. With a small library, the disk space, on modern machines, won't matter if the files are copied. However, those with a large library or little available disk space will soon find the duplication disruptive.

Proposed Solution

Use more database? I mean, this is an obvious solution. A database is currently already being used for metadata and other elements. Of course, I also think that Calibre might also be editing the files as well. In this sense, copying the file for modification in case something bad happens does make sense. However, I would rather use the home directory, whether Windows or Linux or Mac OS X.

Probably spend some time this weekend going over the code and see what is possible. Might even go to the forums and find out why the current structure is used. If it appears that it can be completed quickly, I might start this weekend and hope to finish up by next weekend.

Calibre Improvements

There are two aspects of Calibre that I would focus on, if I didn't have already too many projects on my plate. However, working on Calibre might be a nice break and get me motivated on working on other Python projects, like Trac or Projile.

User Interface

Given what I've been reading, I'm thinking of various ways I can rape the GUI layout. What I mean to say, is that while I'm not an expert yet, I think this might be a good start. I can already see where I'm confused by the library software, so I can imagine where other people might be having difficulties. Truthfully, I'm thinking something similar to Picasa 3 and a mix of Kindle.

  1. Separating The Components

    Calibre appears to be doing 4 things in the space of one main application.

    • Market Place - You can search for books from a list of existing online stores or search online stores for an existing book in your library.
    • News Retrieval - Similar to Google Reader, except for syncing with your phone or converting to e-book format and syncing with a device.
    • E-Book Sync Service - Allows for viewing the collection on a capable device.
    • Library - The main purpose of the software and the majority of the application.

    I would want to make the separation more clear and remove the icons at the top. I would also default to the library, but hide the other three sections / components listed above where others can get to them when needed.

  2. Improving the Library

    The main actions of the library is adding books, converting books, editing books and removing books. The view is displayed by various predefined, what I'll call, directories. The other problem is that while a lot of the functionality is geared towards searching and minimizing the amount of books on display, it doesn't work like a computer or at least is missing normal directory hierarchy. At the very least, a normal directory structure needs to be added to manage what people might already had organized their e-book library.

  3. Removing Books Overload

    The other part is why does the button "Remove Books" even exist? The delete button or the right click and "delete" option has the convention of being used for this action on every piece of software. In fact, such a right click menu already exists and simply has the same options. I suppose it makes sense, they are trying to make it as simple as possible, but by doing so are complicating the system. Delete should simply work as people expect and you should get out of the way. Much of the functionality could had been wrapped into an overall search and book selection that allows for a simple and single "delete" button press or right click and choosing "delete" on the menu. No submenu, just "delete". The right click option is there just for formalities, the expectation is that it should never really be used.

  4. Adding Books Overload

    Same problem with "Add Books". You can simply drag a file or selection of files and drop them into the library and it will add the books. What is "Add Empty Book" and why is it needed? I suppose it was something someone wanted at one point, but when am I going to use it? Adding esoteric features to the application doesn't help with the ease of use. Why didn't the person simply create a txt file, name the file and then simply add it? No matter.

    The real problem is that I have no idea if the directories I add will continuously be synced with the application. Meaning, if I add an e-book that I bought from a web site and download it to one of the directories I added to Calibre, will the book show up? Without checking the source and from other hints from the application, I'm going to guess not. It begs the question, "Why not?" I want this to work like Picasa 3, where I'm not adding books, I'm adding directories which are synced with the application every so often, so that when I add a picture for Picasa 3, it shows up and is synced online. I want Calibre to work like that.

    The way I would improve the "Add Books" functionality is let you choose which directories you want Calibre to sync. Like Picasa 3, allow you to deselect subdirectories and sync down subdirectories automatically. The assumption would be that filenames or titles that match are the same book removing that feature completely, but allow allowing you to correct the libraries assumption.

  5. Cataloging Existing Books (Hard Copy)

    I believe the reason for allowing to add empty books and by ISBN is so that you can list and catalog existing books and display it to people. I suppose to allow them to "check out" the book from you and keep track. However, I doubt this feature is implemented, yet.

    I would improve this by simply allowing you to create a new entry and attempting to download a book picture, if possible. I would check for free ISBN and title book look ups. Since Ebay and other applications do it, it shouldn't be impossible.

  6. Unified Search

    Like Picasa 3, I would put the search on the bottom and automatically select based on criteria. Allowing for managing, hopefully, more easier. Well, technically, the search bar will be at the top and the filtering will be on the bottom.

  7. Book Display

    I rather like the idea of displaying thumbnails and as the default. I think it is wise to allow for displaying a list, I guess sort of like iTunes. Well, I suppose this would be one preference feature left up to the user. Well, actually, a couple of iTunes features are part of the library. I'm unsure what is better verses Picasa 3 over iTunes. Regardless, I would want to do some testing verses the two and sort of find what works better.

  8. Separate Application for Content Server

    Or at least a less visible icon.

    I would rather move the content server code outside of Python into a C#.NET application or potentially a D application. I would rather use the .NET framework since it would allow for utilizing the Windows APIs better. I would then write a Linux Daemon in another language, probably D and attempt to do the same for the Mac OS X with either D (preferred) or Objective C. Even if all the Windows Service and *nix daemons do is execute Python to run the code for all of the platforms.

    I would also move the "Connect" to a device sync functionality or make the content server a "device" as well.

  9. Modifying Books

    There will be a mass edit feature that will be hidden most of the time. I do think that perhaps it will be difficult to use. I'll try an icon for mass editing that opens a pane for changing certain details. Editing a book's metadata or information will be similar to Picasa 3 where it will be a new view. The directional keys will allow moving between books.

    Actually, I'm starting to think I'm overthinking this. In some ways, it is just as simple as it needs to be. I'm just not satisfied until my tongue is so far up its asshole that it is coming snakes.

Maybe a part 2 later. Maybe.

Usability Testing Software Development

The idea has been forming in my mind for a while now and I'm unsure if a project exists yet or at least one that is free. The idea is that there are laser usability testing and plain video testing. What about web cam usability testing? I know the precision will be less than laser, but if you want the El Cheapo brand usability testing, then it is better than nothing right? All you would need are:

  1. Buying a Web Cam or two. Kind of a requirement if I'm going to be doing testing and development. I can't blindly code this.
  2. Facial Recognition

    You do need to know the placement of the eyes. Also if the head moves as it is prone to do, then you need to be able to tell.

  3. Calibration

    I'm guessing is a lot less tricky when dealing with math and precision to simply tell the viewer to stare at a location on the screen for a moment and get the five or more points on the screen. This will give you a sense of where the edges are and the center given the position of the head. May have to "Lock" the head in place. The only problem is that it is easier to tell if the head moved directional but harder (I think) when dealing with depth. I'm thinking that depth could be calculated based on the width of the face, the other problem is telling if the head is turned.

  4. Trigonometry

    The key is transforming the 2D space into a 3D space and back to a 2D space. The camera to the eyes and back to the screen. This creates a triangle which should correlate to a position on the screen. It should be possible to calculate the position by having the person look at the screen. If you know where the person is looking at, it should be possible to calculate the distance of the camera to the screen. Even if it isn't possible, then it doesn't really matter. By getting the points of the screen, you can get the position of the screen based on where the eye might be looking. By sampling the head, the ears (if visible), the nose, the chin, mouth and eyes to get the position of the head given the screen.

    Sure, it is probably cheating, but if it is accurate enough and fast enough, then I don't think any one is going to complain. I'm definitely not going to pester the person to get a ruler or tape measure to measure the web cam to the center point. However, it might be an option if precision is a factor.

  5. Accessing the Web Cam

    I've learned that you do this by DirectShow. This might limit the amount of programming languages and Operating Systems. I'm definitely not going to support Linux, but I may look into Mac OS X support. It will be interesting to find out how to program for that.

  6. Accessing the Screen Content

    The key is to let the person running the usability testing to know where the person was looking and running the test back with what the person looked at and when. My goal also is to let the person know where the mouse went as well.

  7. Play Back

    Well, it isn't useful if it isn't recorded. What I might attempt to do is record the screen along with the user and do post-processing with what the user might have been looking at. This would allow for improving the algorithms for facial recognition and precision of where the user might be looking at.

  8. Research and Development

    It would be fun I think to work on the AI algorithms and math. I'll be brushing up on my trig skills and hope to hone my math ability during this project. I'm hoping the precision isn't terrible and that this can actually be used for what I'm wanting. If not, then I might fall back to taking pictures on an interval and screenshot too. Not going to be as precise as a video running 30 frames per second, but I think it might be much easier to track real time if it was only a single frame a second. Also, the resolution of most web cameras are higher than the video resolution which should hopefully allow for even higher resolution.

Why I Probably Won't Do This

The damn complexity of it all.

The facial recognition will be hard enough and I will work on that first and work on the math for the depth and what not. I'm hoping that it won't be as difficult as I'm fearing. If I can work out the head position when it moves in relation to the camera and, or screen then I'll be at least part way there. It'll be fun I think depending on how long it takes to implement the facial recognition.

I'm also hoping to remove a lot of the tricky Trig and simply have guess the position based on the calibration I'll have the user do beforehand. I'll probably develop a overlay for testing the software to display where I think the user is looking at. This will also help with testing the precision. If the precision is enough that I can tell apart the word, 'it' from the 'i' and the 't' then I think it will be good enough for production. If I can half ass the Trig and still get the precision within the requirements then it will be a win-win. I will still have to spend some time learning Trig and other math subjects to prepare.

Patents. I know they must exist, but I'm not sure what they are. I'm merely guessing that since Photoshop and every camera maker out there can intelligently locate the face that at least that part is patented. I'm also guessing that this idea isn't new and companies have developed technology implementing this idea. Trending on that money maker probably is going to get me sued. Regardless, I'll still attempt to the best of my ability to create this and research existing patents as I develop. Perhaps if I'm super nice, I'll be able to use the patents without incurring a bankrupting huge cost.

I'm super lazy and I have another 3 or 4 projects I want to implement. Depending on my motivation, I'll probably work on them first.

Why I Want To Do This

Well, Selenium is great for browser testing. However, while that is great. I think it will be equally great if someone would create an easy to use User Usability Testing software like they use in labs, but far cheaper. If the goal is to do usability testing often, then what better way than using cheap equipment and a whole lot of people. If you could use anyone with a web cam, then you could even do so remotely.

I did have plans to somehow have the developer see the user as the person is doing the usability testing. I think that going through Skype or uploading the finished product should be sufficient for the amount of effort and time I'm planning on putting into this project.

Of course it will be open source and of course I'll probably be violating a crazy amount of patents to implement this. The later part does kind of suck, since I will lack the funds to license any patents I'll be infringing on and also that there are damn patents in the first place! Ah well, I'm hoping that the former will at least allow a better programmer to come along and use a method that isn't patented.

Motivations for Programming

My motivation for programming the Minecraft Proxy will probably come when 1.9 comes out and I blow away my world. As long as I spent on that world, I'm not going to have a lot of motivation unless I have some intelligent helpers. I'll probably be spending some couple of months working on the project. I keep itching to do the networking code and seeing how well that is handled. Motivation for programming Python is also high. I have a project I wanted to have completed by the end of this weekend and I haven't even started. I still think I can complete it, but the code will be sloppy, most likely needing rewrites and well, it will need to be rewritten anyway. I'm thinking about taking the Projile project in stages. I'm also thinking of developing my own framework starting with the controller, but leaving the models and views to third-party libraries. The controller will probably take a week or two, depending on how well the initial code is developed and what existing framework I want to closely match.

My goals for the Python Controller Framework is similar to the Minecraft Proxy. I'm not going to focus on programming either project the "correct" way given the idioms of the programming languages. The code most likely will be developed similar to the patterns I've learned from PHP and Java. Which probably won't be all that bad. I'm going to take "Mythical Man-Month" approach and toss the first implementation as not really a prototype, but an solution that will hopefully lead to a better working solution. Similar to the Projile project itself.

The goal for the Framework is to create a configuration-less Controller framework that handles installation, upgrading, plugins and authentication. The authentication I want to support many different OpenID services and probably will write a Facebook Python framework, unless one doesn't already exists and is on par to what I'm looking for. I'll probably write one anyway, since I want to learn the inner workings of Facebook Open Graph or whatever the public API is called.

From my readings of Pragmatic Programmer and "Your Brain at Work", it seems that the best way to be motivated for a project is to take it in steps. To focus on an achievable goal and work towards it. In the case of Pragmatic Programmer, if you find that the goal is not exactly what you've been looking for, then maybe the entire solution wasn't great in the first place and not much is lost.

The other part of this is to develop components that are separate. I'm not sure how I'm going to do that with the Minecraft Proxy in D. The Java code is... not exactly decoupled and I doubt the D code will be either. I'm thinking the Python framework code will be decoupled enough to start work and should be able to be refactored to its separate components.

Of course, while thinking about doing this, I should just be doing it. The problem is that the more I think about the project and why I'm not doing the projects the better the solutions. Another demotivating part is sucking at design and usability. Even with the books I've been reading, it is still extremely difficult. Of course, I haven't actually read any basic design books. That will be my next stop. I think I've had enough of the basics of usability and need to focus on design. Once this roadblock is dealt with, I'll hopefully be more willing to focus on this project.

It is holding up long term projects, which is further motivating me to want to start working on it. I keep thinking more and more about the farmer sim browser game and how fun it would be to finally be able to play it. The kinds of stuff I want to do! Regardless, I'm kind of hoping Projile will be the foundation for developing the games and learning better programming and usability.

Dealing With People

The past 3 weeks has been busy with listening to books. I've listened to "Coping with Difficult People" by Robert Bramson, "21 Ways to Defuse Anger and Calm People Down" by Michael Staver, "The Believing Brain" by Michael Shermer, "The Five Dysfunctions of a Team: A Leadership Fable" by Patrick Lencioni, "Your Brain at Work" by David Rock, "Verbal Art of Self Defense" by Suzette Haden Elgin and "How to Win Friends and Influence People" by Dale Carnegie. I've noticed some similarities with some of the subjects with the books. The book, "Your Brain at Work" discusses at one point the anger people have and "21 Ways to Defuse Anger" actually discusses and deals with that subject a bit more. "Your Brain at Work" also discusses some of the ideas further emphasized in "The Five Dysfunctions of a Team" and quite possibly "How to Win Friends".

It is interesting to me how there is some overlap with the books, where one book does not explain the brain fundamentals in "21 Ways" and "Five Dysfunctions", it is explained in "Your Brain at Work". It is helpful to fill in the gaps and relate the topics together. Interesting "Brain Bugs" does also help with the inner workings as well explaining how the brain works or doesn't work.

What is interesting is that with all of these books with dealing with people, the common thread is that you don't deal with people by being mean or being a jerk. This causes the opposite affect as the brain rejects the idea as it is triggering its threat response. The best way to deal with someone is by getting their dopamine levels raised. You can train someone better with treats and praise without them even realizing what you are doing.

Not quite manipulation, since from some of the books, people can tell when someone is faking. Much like when people realize someone isn't truly smiling. The brain knows what people can't quite put in words triggering a response that isn't normally favorable to the one attempting to do the manipulation. Of course, this does work, but generally triggers the threat response without the person realizing why.

My normal response has been, "Tough Love" and I'm realizing now the reason it has never worked. I simply thought, incorrectly, that if they knew how right I was and competent and logical my solution was that the person would understand and follow through with my plan. However, it is the way you say things that drive people against or for you. The problem, I suppose, is that it takes a lot of practice and while I'm putting some of the ideas into how I deal with people, I still tend to revert back to my asshole self.

People are motivated by many things and I think the "Verbal Art of Self Defense" says it best, "Assume that the person says is true for them." To often we don't do that and it leads to misunderstandings.

It is frustrating when you are talking to someone and they (you think) assume that you are an idiot and reject what you are saying as wrong based on the premise of missing information. Instead of questioning their train of thought and asking for further clarification, the thought patterns leads down one of trolling. You end up thinking the person is unfair, since the person never let you explain your idea and kept insulting it and they simply continue thinking you don't know what you are talking about.

Spin this around and applying the Verbal art of Self Defense, we can probably apply some of the rules from 21 Ways to Defuse Anger and approach this conversation a different way that leads the person to understanding your train of thought and accepting your idea. It is difficult. "Your Brain at Work" does discuss many techniques for how to remain calm and continue thinking with a cool head, but it takes practice. It takes never being angry, because anger prevents the thought flow. More than that it takes working through your anger so that it doesn't simply remain in the background waiting to boil over. It is very difficult.

To often I remember becoming so angry that I couldn't think straight. Of course, both "Your Brain at Work" and "21 Ways to Defuse Anger" explain why, more so in "Your Brain at Work" and alluded to in "21 Ways".

It is interesting that by listening to these books after each other that it makes more sense. The Verbal Art of Self Defense and 21 Ways to Defuse Anger talks about practical ways to handle situations, however understanding the underlying reasons why your brain acts the way it does helps too.

I find this topic interesting and it is nice that these books are short and are, I think, made to be listened to over and over again. Which I think is how the ideas really start to stick. Applying maybe one or two techniques from each book will probably be better since you will be better off as a person and in relationships. I also think that the full wealth of the books will be researching them as you would a bible until the techniques and information in the books or audio listens become second nature.

It is difficult to change one's behavior. I know this intimately well. Taking baby steps seems to be working well for me and I look forward to hopefully being in enough situations where I can learn and apply more of what I learned from these books. It is a good thing I think to never be in a situation to need Verbal Art of Self Defense or 21 Ways to Defuse Anger, but also bad since you'll never have the apply the techniques in order to drill them into your mind with practice.

Minecraft Proxy in D

The Minecraft Proxy in D will probably be started within the hour. I'm going to do some minor research and then jump into some test code and then some network code. My goal is to have a connection to a Minecraft 1.8 server running and have a miner running around mining blocks and laying track. Oh yeah, I also want to implement contract programming and unit tests. It will slow down the development, but I think it will hopefully limit the amount of debugging and problems I'll have during development.

  1. Compile First D Program: Hello World

    Have to start somewhere. I figure if I have this done today or within an hour, that I'll have motivation to continue.

  2. Minecraft Protocol Module

    There probably is an existing implementation in D, but I think it will be helpful to write my own module. I need to write the code to be thread safe, but I don't plan on implementing the network code using threads yet. I won't have the benefit of a proxy through Minecraft software, so I'm instead going to have to implement the server authentication scheme. That is getting the authentication string and passing it to the server. This will take some time and testing.

  3. Network Code

    I'll use the completed Minecraft Protocol module and attempt to implement the network code. I hope I don't have to implement threading, but remembering the Java experience, I'll probably do it anyway.

  4. Minecraft Chunk Module

    I'm kind of hoping that I won't have to require a database and I want to see how much space is taken by the D implementation. Should be fun and interesting. I'll probably use a multidimensional array for chunks and coordinates. Should be simple, but probably won't be efficient. I'm going for simple and quick. I'll refactor the code later to be more efficient, but meh, it might turn out to be efficient enough that I won't need to use something else. Of course, if I can't implement the array in this way, then I'll have to come up with another design.

  5. Minecraft Entity Module

    The problem and what I'll need to implement to create some sort of intelligence is to track entities. This is something, I don't currently handle and for what I'm planning I'm going to need to do it. The problem so far is that I don't know where monsters, mobs and other players are. I need to be able to track this information to ensure that my code correctly handles each mob or player.

  6. A* and Dijkstra Pathfinding Module

    I had forgotten that this was a requirement. Oh well, I think I understand the A* and Dijkstra pathfinding algorithms enough now that this shouldn't be that difficult. Or I'll simply brute force it. I'll probably require this be a separate thread, crap now I need to solve many different thread and dead locking with the Chunk Module. It is possible that I could have this part of the Chunk Module and simply spawn a new thread and when it completes pass it back to the caller. It depends on where the pathfinding will be.

  7. Intelligence Module(?)

    I'm debating this. Creating a Fuzzy Logic module could take anywhere from half a day to 2 weeks. I'm not sure what benefit I'll get from it either. I want to do it, but I'm kind of hoping that I can kludge my way through some sort of algorithm that sort of doing something similar.

    I don't know, it will be worth looking into implementing it, since a lot of the code I'll be writing will mimic fuzzy logic. Quite possibly I'll be saving some time instead of testing and developing it the boolean way. I don't know. I think I'll head down the path of not implementing true Fuzzy Logic module and instead focus on different actions.

    The main loop will check different conditions. Is there a mob within 20 meters? What type of mob is it? Does the mob have a direct line of sight? What is the players health? Is there a weapon available? Meh, it will probably be worth investing the time implementing fuzzy logic just so that I'm not writing a ton of code that will be damn near impossible to debug.

  8. Behavior Module

    My goal ultimately is to separate tasks into behaviors and then into sub actions using the intelligence module. In this way, I'll be switching between behaviors and focusing on completing the task.

  9. Chat Module

    I'm not going to be handling this intelligently. I'm simply going to pass commands to the player using chat commands. Therefore, the code needs to be able to parse the chat and pick up on the commands. At some point in the future, I may extend the code to intelligently talk to others using the chat dialogue. Since majority of the players will be on my own personal server, I don't really see a need to spend a lot of time on this.


  • Mining

    Majority of the time will hopefully be spent on this behavior. Mining will focus on precious resources and then collecting other resources depending on different needs. Like for building a chest might switch from one resource to collecting wood instead.

    The Mining action requires that the player know about lava, how to avoid lava, how to mine around lava and how to move lava. I think that since the player has a lot of time, that I'll instead have the player move the lava to an area where there is an infinite water pool for pouring the lava and turning into obsidian. When the water pool is filled, then the player will mine the obsidian and put it away.

    My plan is basically to have the player mine the bottom 40 block layers everywhere. But not the ravines or new places. At some point, I'll think of a plan for what to do with the soon to be large underground lair. I think it would also be funny to join a server and after a week or so, be like, "WTF?" I don't know. I could go many different ways with this.

  • Crafting

    Need to set the player code to do this whenever it is needed. Mostly what will be crafted are chests, furnaces, rail, powered rail, mine carts, chest mine carts, workbench, tools and weapons.

  • Farming

    Food is important, so at certain points, the player's character will go out and spend an hour or so collecting food by farming. Some actions might be to collect seeds, mushrooms, plant mushrooms and use bonemeal (if available).

  • Building

    At some point I want to develop the player to build rail stations or maintain current rail stations. I also want the player to build different architecture for the rail ways and maybe light houses and walls. This will also include farming land in such a way to enable efficient and maybe pretty solutions.

At this point, I'll probably simply implement just enough of the Crafting and Mining in order to find resources and not run out of food, tools and weapons. I'll tell the player where to go and what to do using signs, which should help the player not do something that I'll regret.


The application won't just be AI player. I'll eventually implement OpenGL display from the Java version to the D version using a library. I also want to implement a GUI using another library.

I think that if after this weekend I have the network code and minecraft protocol code working. I'll be good. Probably spend a few weeks or months away from the code depending on motivation and finish up the other plans later. If I do get to the pathfinding parts, I think that will be fun and if I can implement some primitive mining code, that will be even better. I'm optimistic I can do the protocol and network code this weekend. Not so much all of the other stuff. I'll really only have a day and a half work on this project, so not a whole lot of time. I'll spend a day and a half working on the Python Web Spider project and end the weekend.