Miles Johnson

Security::cipher() problems and Suhosin

Over a week ago I migrated one of my sites to a MediaTemple (ve) Server. I installed nginx, PHP 5.3.2 (with Suhosin), MySQL, APC and Memcache. After a little configuration and debugging, everything was working smoothly except encrypted cookies. I was using my AutoLogin component and login never persisted, but regular cookies worked correctly. I thought it would be an easy fix but I was wrong (I will post another blog on this topic).

While trying to figure out the encryption problem, I debugged the CookieComponent. I tested each function individually around a fake array. I tested __implode(), __encrypt(), __explode() and Security::cipher(). My results were all over the place. When I tested the script locally, everything worked correctly and the ciphering process worked as it should (below). But on the live server, all the encryption strings would change each refresh and the un-cipher would never work.

Implode:
username|gearvOsh,password|dsadnas79dn7n1279dnadnadb1p2sdasd,hash|761c1168c645784b78f5f967160ab390,time|1286152916
Ciphered:
z~¡|ßÛí�"� =�'¥¾E¨ÿÏy� íåÇv¸”^ëS ¦ì•ˆxuŽ?��ãbm}ÜfÇ“„bËV´w±(ÐU�"éQ’Qñn²^6¿2r‰½ùé‹ÄáÄ,‡ ›d¤<ó`'ëÆìÏg$V")ËdÁ&µ‰
Base64 encode:
Q2FrZQ==.en6hfN/b7dMfij3Spb4ERaj/z3ng7eXHdriUXutTIKbslYh4dR6OP8bjYm193GbHk4Riy1a0d7Eowx+QVZbpUZJR8W6yXja/MnKJvfnpixsIxOHELIcgm2QHpDzzYCfrxhrsz2ckVggijwcpy2TBJrWJ
Using Cookie::__encrypt():
Q2FrZQ==.en6hfN/b7dMfij3Spb4ERaj/z3ng7eXHdriUXutTIKbslYh4dR6OP8bjYm193GbHk4Riy1a0d7Eowx+QVZbpUZJR8W6yXja/MnKJvfnpixsIxOHELIcgm2QHpDzzYCfrxhrsz2ckVggijwcpy2TBJrWJ
Base64 decode:
z~¡|ßÛí�"� =�'¥¾E¨ÿÏy� íåÇv¸”^ëS ¦ì•ˆxuŽ?Æãbm}ÜfÇ“„bËV´w±(ÐU�"éQ’Qñn²^6¿2r‰½ùé‹ÄáÄ,‡ ›d¤<ó`'ëÆìÏg$V")ËdÁ&µ‰
De-ciphered:
username|gearvOsh,password|dsadnas79dn7n1279dnadnadb1p2sdasd,hash|761c1168c645784b78f5f967160ab390,time|1286152916

As you can see from the debug (on my local machine), the beginning and resulting strings are the same. I then found out the problem was with Security::cipher() and srand(). I later found out that Suhosin was overriding the srand() and mt_srand() functions, which was causing my seed to always change. I tried disabling Suhosins overwrites, but to no avail.

// Did not work
ini_set('suhosin.mt_srand.ignore', 0);
ini_set('suhosin.srand.ignore', 0);

After some googling and a lot of frustration, I finally found a solution. Thanks to Edgar Valarezo, I was able to get my encrypted cookies working properly. However, I had to overwrite CakePHP's core Security class and will need to maintain it every time I upgrade Cake. Here is the new cipher() snippet for your convenience.

function cipher($text, $key = '') {
    $key .= Configure::read('Security.cipherSeed');
    $out = '';
    $textLength = strlen($text);
    $keyLength = strlen($key);
    $k = 0;
    for ($i = 0; $i < $textLength; $i++) {
        $seed = md5($key . $key[($k++) % $keyLength]);
        $mask = hexdec($seed[6] . $seed[9]); // :)
        $out .= chr(ord($text[$i]) ^ $mask);
    }
    return $out;
}

I see this as a temporary fix and would love any information on this topic. If you ran into this same problem, how did you correct? Is it possible to seed correctly with Suhosin? Thanks again!

Single or multiple feed aggregation?

I recently published a beta version of my feeds plugin for CakePHP. This plugin was previously the FeedAggregator component, but it made more sense to break it up into a datasource and model, and finally package as a plugin. The datasource is pretty straight forward as it accepts an array of URLs (RSS feeds), fetches each one through an HTTP request, parses the XML into an array, and then returns the result. The model is simply there for your convenience.

Now the dilemma I am running into is whether or not the datasource should only parse one feed at a time or multiple feeds (currently this). It can go either way: the datasource parses multiple feeds and uses the model to return them, or the datasource parses one feed and the model manages multiple connections and merging. Now the big question for you guys... Should the datasource parse one feed at a time or multiple feeds?

Currently you use the model to pass an array of URLs (through the conditions option), the limit, which fields (elements in the XML) you want returned, and some cache/feed settings. Here is a quick example:

// Multiple feed parsing
$feeds = $this->Aggregator->find('all', array(
	'conditions' => array(
		'Starcraft 2 Armory' => 'http://feeds.feedburner.com/starcraft',
		'Miles Johnson' => 'http://feeds.feedburner.com/milesj'
	),
	'feed' => array(
		'cache' => 'feedCacheKey',
		'expires' => '+24 hours',
		'explicit' => true
	)
));

And I am assuming single feed parsing would look something like this:

$feed = $this->Aggregator->find('first', array(
    'conditions' => array('http://feeds.feedburner.com/milesj'),
    'feed' => array(
        'cache' => 'feedCacheKey',
        'expires' => '+24 hours',
        'explicit' => true
    )
));

I am kind of split on how I should go about this and would really love your opinion. I am currently leaning towards multiple feed parsing (current implementation), but if someone has a good argument in not doing so, I will change it.

Making sure debug is off in production

Over a year ago I wrote about turning debug off automatically in production. That post I wrote is completely wrong (to an extent). The theory is correct but the execution was incorrect. Even one of the comments pointed out the problem, but I haven't had time to blog about it till now.

About a month ago I realized my implementation was wrong when one of my live sites was outputting MySQL errors and database information (including passwords) to all my users. Since debug in core.php was set to 2, and then disabled to 0 in bootstrap.php, the errors were being triggered before bootstrap was loaded. This was a huge problem as it printed out vital DB information.

It is an easy fix however, simply switch around the values from my previous entry. Debug in core.php should be set to 0 and in bootstrap.php it should be set to 2! That fixes the startup errors that appear before the bootstrap process.

if (env('REMOTE_ADDR') == '127.0.0.1') {
	Configure::write('debug', 2);
}

The widely unused and powerful setAction()

It has been quite a while since I last developed in Cake, but earlier today I spent a good 5+ hours working on some new controller internals for one of my apps. While working on this app, I thought I would give the not so used method, setAction() a spin. If you are unfamiliar with setAction(), it's a method that will internally forward one action to another. This is incredibly useful as it allows you to switch actions without having to do another HTTP request, nor having to reload all the controller and related objects; could be a huge saver on overhead.

Here's a quick example of a regular action:

public function profile($id) {
	$user = $this->User->findById($id);
	if (empty($user)) {
		$this->redirect(array('action' => 'listing'));
	}
	$this->set('user', $user);
}

What the action is doing is relatively simple. First it grabs a user based on an ID, checks to see if the result is empty, if so redirects to the users listing page, else sets the data. You could either redirect the action like I have, or you can simply throw an error message in the view saying something like "No user was found with that ID". I personally prefer redirecting in most cases, but it's a matter of preference. Lets move forward and try setAction() now.

public function profile($id) {
	$user = $this->User->findById($id);
	if (empty($user)) {
		return $this->setAction('listing');
	}
	$this->set('user', $user);
}

As you can see, I simply switched the redirect() with setAction(). Now if we visit a profile with an non-existent user, the listing action will be rendered without having to do another HTTP request. Also, do notice the use of return; this is extremely important because if you do not use return, all the logic after the setAction() will be ran and processed, causing unneeded overhead and unexpected results.

One downside that I have ran into with this approach is that the URL in the address bar does not change. But 99% of the time it won't matter as the page you render in its place will contain links to the correct pages or the user won't be copy and pasting the URL anyways.

Upgrading the forum plugin to 1.8

I have been working on the new 1.8 version of the forum for almost 2 months, simply because version 1.7 was completely broken. It used PHP 5.3 constants which didn't work on 99% of the users machines. This problem has now been corrected in the new 1.8 version, but that's not all that changed. All topics, forums and forum categories now use the sluggable behavior for pretty URLs. Furthermore, the Installer has been rewritten completely for faster and easier usage. The main purpose of this post is to direct you in the 1.8 upgrade process as there are many things you need to do.

Overwriting an older version

You will need to overwrite ALL files within the plugin. Basically every view, every model, every controller, etc has been modified in some way; primarily to support the new slugs. Sorry to all those who have changed the HTML.

Patching your plugin

The new installer system creates a legacy config file to be used for future upgrades and to maintain the current installation. If you are doing a fresh install, you can skip this step. Once you have updated all the files, you will need to patch your installation to build that config file. You can do this from within the install controller.

yourdomain.com/forum/install/patch

The patch process will require 3 things: the database config you want to use, the table prefix value (if you have one) and a checkbox that determines if you used the plugins native user table or a pre-existing one.

Upgrading the database

With the new slug changes, it requires you to upgrade your database tables with the new column and all rows to be populated with their slug equivalent. This process takes 2 steps to achieve, the first having you run an SQL command from within the installer. You can access this process at the URL below.

yourdomain.com/forum/install/upgrade_1_8

The second step requires you to use the Cake console and to run the slugify shell. This shell will populate all the empty slug rows (it can be used any time, not just the upgrade process). Your shell command may look like the following.

cake -app /path/to/app slugify

Fresh install

If you are running a fresh install, the process is extremely simply. Just go to the install controller, hit the "Begin Installation" button and follow the steps. The process will check your database for conflicts, create the tables, patch the install and create the admin user. I tried to make it as simply and straight forward as possible.

See if it's working!

Whether you did a fresh install, or patched an old one, the last thing to do is actually use the forum. You should be checking to make sure all the slugs were generated, the pages work, the links link correctly, so on and so forth. If you find a bug or problem, PLEASE report it to me so I can fix it.

Enjoy! (Sorry for the long wait time)

Setting up cron jobs with Cake shells

This post will primarily be about the process I had to go through to get cron jobs and shells working on Dreamhost. The main problem I kept running into was that the shells would simply not work at all through a cron job; yet they worked when I manually did it through SSH. The weird thing was the source of the console/cake file was being printed in my logs, instead of executed. Below is a quick checklist of things to look out for:

TERM environment variable not set

This isn't really an error, but the TERM variable is used by the server environment and does not affect the shell, even if the error is thrown. Now I may not know much about server maintenance and configuration, but adding the following code to console/cake seemed to do the trick (Hats off to Matt Curry for the tip). And of course, you should change linux to whatever your server is.

TERM=linux
export TERM

Cron environment not PHP 5

This was a weird problem I ran into. It seemed the environment in which my crons ran was PHP 4, so I was receiving unexpected results and would receive this error (it mainly depends where the php folder is located and if its loaded correctly).

console/cake: line 30: exec: php: not found

To fix this, I had to rewrite the cake console file and update it with the path to the PHP folder (path should represent your servers structure). You should only need to do this if you can't upgrade PHP to 5 for some reason.

exec php -q ${LIB}cake.php -working "${APP}" "$@"
# change to
exec /usr/local/php5/bin/php -q ${LIB}cake.php -working "${APP}" "$@"

Making sure your cron has ownership

A simple problem with a simple fix. Since the user that uploaded the console files was different than the user running the cron, I would receive this error:

sh: cake/console/cake: Permission denied

All I had to do was switch the cron user to the user that owns the file, and it fixed the permissions problem.

Making sure the cake file uses unix line endings

This was another pain in the ass that took me forever to get working. My cron environment would throw this error once I fixed the things above (this applies to my Dreamhost server and may not apply to all).

sh: cake/console/cake: /bin/bash^M: bad interpreter: No such file or directory

It required me contacting my host and asking for help on this one. I tried saving all my files as unix and made sure my FTP was not converting them in any way. Still nothing. So my host told me about this SSH command called dos2unix which would convert the file to unix line endings, and wallah, magic! Everything seemed to be working now.

dos2unix cake/console/cake

These are just a few of the minor setbacks I had to deal with when setting up my cron jobs. Most of this really applies to my Dreamhost server, as my Media Temple server worked right away with no configuration. Furthermore, here are some more Dreamhost articles that I used for this problem, that might be of aid:

Editing Your Environment Profile
Make PHP5 the Default in the Shell

Pitfalls within Cake's Cache engine

For the past week I have been working on a DataSource for the WeGame API. Everything was working fine until I added some logic to cache the results. No matter what I did I received the following fatal error:

//Fatal error: Call to a member function init() on a non-object in C:\xampp\htdocs\cake_1.2.5\cake\libs\cache.php on line 195
// Line 195
$_this->_Engine[$engine]->init($settings);

The error was being thrown from within Cache::set() and it seemed to fail because the actual storage engine object was not instantiated. I spent a good couple hours trying to track down the bug, but to no avail. I tried removing Cache::set(), placing Cache::config() within config/core.php, but still nothing.

Finally I thought to myself "Let me check my Config settings", so I did. I noticed that Cache.disable was set to true, and once I set it to false the whole system worked! Bah, pretty annoying right? I sure love Cake, but its error and exception handling isn't that great. I've ran into so many of these case scenarios where a simple trigger_error() or uncaught exception would of helped (This is for you Cake team *cough*2.0*cough*).

Another thing I would like to note, is that the path option in Cache::config() does not actually create the folder, you must do that manually. Here's how I resolved that problem.

$cachePath = CACHE .'we_game'. DS;
// Create the cache dir
if (!file_exists($cachePath)) {
	$this->Folder = new Folder();
	$this->Folder->create($cachePath, 0777);
}
Cache::config('weGame', array('path' => $cachePath));

Retrieving tables within your Cake database

I have been working lately on adding an installation process to my forum plugin. For this process I needed a way to retrieve a list of tables within a selected database, to make sure there are no table name conflicts. After asking the question on Twitter, I got the answer thanks to @pierremartin and @mark_story (of course!). You simply use the ConnectionManager class.

$db = ConnectionManager::getDataSource('default');
$tables = $db->listSources();

Very handy, and a lot easier then writing a custom query with execute().

The flawless upgrade to Cake 1.3

Cake 1.3 has been around for quite some time now, but I never had the desire to upgrade with it being in such a beginning stage. Well seeing as how its now in Alpha and many users are testing it with no problems, I thought it would be a good idea. The project I was going to upgrade is quite large, heres a quick rundown of just how large:

19 controllers, 8 components, 15 locales, 64 models, 2 plugins, 6 vendors, 8 stylesheets, 14 javascripts, 7 helpers, 159 views

This isn't counting the admin panel, which is a whole separate app, and which I have not upgraded yet. Now lets get onto the upgrade, its probably the easiest thing to do. All I did was follow the migration guide. First off was checking to see if I used any deprecated methods, which was simple, all I did was search the whole app folder for words like del(. I was surprised to find I used it like 5 times, when I always preferred the full delete().

Next in the list was upgrading to the new Router conventions, which lucky for me didn't apply. I never used any non basic conventions, I mean why would you! One thing that bugged me though is the new page title implementation. It took me quite a while to go through all those controllers, and all the actions within them. Another nuisance was adding "echo" to all the flash() calls in the views, and removing the "inline" parameter from all meta() and css() calls, but wasn't too time consuming.

One of my most favorite updates was the new pagination support with passedArgs. I had to hack all my paging before, but in the new update I removed it all and it worked wonderfully! Thanks Cake team :]

All in all, the upgrade went smooth. If you follow Cake conventions all the way through, you don't need to do much alterations. I for one am looking forward to the future versions.

Fixing the missing Model and AppModel replacement errors

If you are a CakePHP developer, I am almost certain you have run into this problem on multiple occasions. What problem are you referring to you ask? Well, the problem that you have when your model is not being found and Cake automatically substitutes it with AppModel. Cake does this on purpose so that your relations and HABTM's do not need the junction Model to operate correctly. This works in most cases, but sometimes you get some weird errors or missing method problems. But before I continue with this, lets setup a quick scenario so I can better explain this problem. I will be using an example of relating users to teams.

User Model - users Table
Team Model - teams Table
TeamsUser Model - teams_users Table (Join)
User -> hasAndBelongsToMany -> Team
Team -> hasAndBelongsToMany -> User
TeamsUser -> belongsTo -> User, Team

In our code we are stating that a user can be on multiple teams, and a team can have multiple users. Are relation seems pretty simple, but lets not rely on Cake to magically figure everything out. In some cases we would receive the following errors:

• Model "User" is not associated with model "Team"
• Warning (512): SQL Error: 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'myCustomMethod' at line 1 [CORE\cake\libs
  \model\datasources\dbo_source.php, line 514] Query: myCustomMethod

Now what do those errors mean? Well the first one states that our Models can not be related. So the first thing we need to do is check that our Models naming conventions are correct: model names are singular and camelcased, where as the filenames are singular, underscored and lowercased (without having _model in the filename). The second error means we are trying to call a method on a Model that does not have that method in its class. In other words, it basically means our Model is not being loaded and the AppModel is being loaded in its place. This happens when Cakes naming magic fails to load the correct Model or we have improperly followed conventions.

Most of these problems can be fixed by following the proper naming conventions. I'd also like to note that most of these errors appear during HABTM relations, and can easily be fixed with the following:

• With any relation, its best to define the "className" parameter. By doing this, we are telling Cake the exact name of our Model, instead of having Cake rely on the relation name (which can easily be changed to custom text).
• When defining conditions or containments, be sure to include the model name followed by the column name: Model.field
• Important! When working with complex HABTM relations, define the "className, joinTable and with" parameters. If you have complex relations with crazy table names, its highly required you define the joinTable and with parameters.

During one of my projects, I had a Model that had a HABTM relationship, in which that HABTM had its own HABTM relationship, so it got quite complex. I ran into problems over and over again where Models would not be related and the AppModel was used instead. I later found out that the problem was the with parameter. Since I didn't define it manually, Cake could not figure out the correct HABTM Model to use, hence the whole application broke. Its pretty funny that my whole app didn't work for weeks, all based on this small declaration.

So, since we know how to fix this problem, lets define our relations:

class User extends AppModel {
	public $hasAndBelongsToMany = array('Team' => array(
		'className' => 'Team',
		'joinTable' => 'teams_users',
		'with' => 'TeamsUser',
		'foreignKey' => 'user_id',
		'associationForeignKey' => 'team_id'
	));
}

So in conclusion, follow the naming conventions and define the required parameters when creating relations, and I promise you, you won't have any problems.