tag:blogger.com,1999:blog-6455246628790577392024-03-13T14:10:14.711-07:00Halomede BlogTechnical tidbits from our daily experiences.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-645524662879057739.post-26015434284014451892009-12-16T17:41:00.000-08:002010-10-15T15:33:26.544-07:00Technical Analysis of bi-directional Sun IDM and AD password syncingOne of the more common uses for Sun Identity Manager is to sync passwords to an Active Directory environment. If AD is not used as the primary authentication service, you'll probably also need to sync passwords back from AD to IDM, so that users can change their password from their workstations. This is especially important in the case where their password has expired and they can't access the IDM web interface to change it without logging into their workstation. Sun IDM provides methods for this bi-directional sync, but the documentation prior to version 8.1 is very sparse and even in 8.1 it is lacking in some areas and fragmented into several separate documents. This post tries to consolidate the information and fill in some of the gaps.<br /><br />The two additional components that are required for password synchronization are the IDM Gateway, which synchronizes from IDM to AD and PasswordSync, which synchronizes from AD to IDM. Below is a diagram that details how the different components work together and communicate.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwSrWtRhyphenhyphenfiG6L0wlIXZrcPbdv-DkAw1HgGkSPrcLGuZ8-E7sjFHGsyi3sJqPGuNMXyzgAEz6FhLCNLTQbrFpXe_-bgG7AwnKrvFypXjZ2Vvj5WpvdLbY0Gs_2JycuD50-PYsoDC5Etos/s1600-h/IDM_password_sync.jpg"><img style="cursor: pointer; width: 339px; height: 400px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwSrWtRhyphenhyphenfiG6L0wlIXZrcPbdv-DkAw1HgGkSPrcLGuZ8-E7sjFHGsyi3sJqPGuNMXyzgAEz6FhLCNLTQbrFpXe_-bgG7AwnKrvFypXjZ2Vvj5WpvdLbY0Gs_2JycuD50-PYsoDC5Etos/s400/IDM_password_sync.jpg" alt="" id="BLOGGER_PHOTO_ID_5416281146398903298" border="0" /></a><br /><br />Here is a quick list of best practices to follow. For more detail about the communication protocols read on past the bullet points.<br /><ul><br /><li>Change the default Gateway encryption key</li><br /><li>Use an active-standby configuration behind a load-balancer for the IDM Gateway. Active-active is not supported.</li><br /><li>Ensure that you are using encryption between the IDM Gateway and the DCs, either Kerberos or SSL.</li><br /><li>Ensure that you are using HTTP over SSL for the communication from the PasswordSync application on the DCs to the PasswordSync servlet.</li><br /><li>Use a JMS server to handle the PasswordSync events. This will help ensure that no password changes are lost and allow for much better auditing of change events.</li><br /></ul><br /><br />The communication between the Sun IDM application and the IDM Gateway is done over port 9278 using the usual protocol for communication with resource adapters. This traffic is encrypted using 3DES by the RASecureConnection class. This may come as a bit of a surprise, since the AD resource adapter in IDM has an encryption type setting of None, Kerberos, or SSL. This setting is for the communication between the Gateway and the DC, not between IDM and the Gateway.<br /><br />Since 3DES is a symmetric protocol, it is important that you change the key from the default key that is compiled into the application. You can do this through the IDM admin interface, Server Tasks -> Run Tasks -> Manage Server Encryption and select "Manage Gateway Keys"<br /><br />Password changes initiated through standard methods in AD (i.e. administrative resets, user-initiated changes) are sent to IDM through the PasswordSync application. This application is installed on the DC and configured through the user interface as detailed <a href="http://docs.sun.com/app/docs/doc/820-5822/byavb?a=view">here</a>. It communicates with the PasswordSync servlet, which is shipped with IDM and by default runs as part of the IDM web application. However, it can be split out into a separate servlet container. There are two options for the communication between the PasswordSync servlet and the IDM application. The simpler, but less reliable option is direct communication to the IDM application, the other is to use a Java Messaging Server (JMS), which the IDM application will interface with. This will allow for queuing, guaranteed delivery of the password change messages and better reporting and logging. The JMS method is preferred, as without it, password changes may be lost if there is a problem with the IDM application.<br /><br />You will need to ensure that the PasswordSync -> PasswordSync servlet communication is encrypted via SSL. You will need to import the SSL certificate that the PasswordSync servlet is using onto the DCs as documented <a href="http://support.microsoft.com/kb/816794">here</a>. You will also need to remember to update it on the DCs whenever the certificate is changed or renewed.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-12571590001410187462009-10-23T18:56:00.000-07:002010-10-15T15:34:04.706-07:00Running Sun Directory Server 6.3 as a non root userIt's a standard best practice to run services with a dedicated user if possible. Sun's Directory server Enterprise Edition supports this, but, as with many Sun products, the documentation on how to do so is scattered in many different places. Here's a quick how-to guide on to do it for DSEE 6.3.1 on Solaris 10. This only covers the Directory Server component, not the Directory Proxy Server or Control Center components. But don't worry, more posts are in the works for those components!<br /><br />These steps presume that you have already installed the DSEE software.<br /><br />Step 1, create the user. Here's quick instructions to create username "ldap" with group "ldap":<br /><br /><pre class='postCode'><br />groupadd ldap<br />useradd -g ldap -s /bin/false -c "ldap" ldap<br />usermod -K defaultpriv=basic,proc_owner,net_privaddr ldap<br /></pre><br /><br />If you're not familiar with Solaris's RBAC (Role Based Access Control) , the last command grants the ldap user the privileges to bind to a privileged port number (net_privaddr), which is needed to use the standard LDAP ports of 389 and 636. The proc_owner command allows the ldap user to send signals to other processes as well as those that it owns.<br /><br />Step 2, create the directory server instance.<br /><br /><pre class='postCode'>dsadm create -p 389 -P 636 -u ldap -g ldap /opt/ds_homedir</pre><br /><br />Where the last argument is the directory that you want to use to store the configuration and data for your new instance.<br /><br />Step 3, enable the service within SMF. This has two functions: first, it will make sure that the service will restart automatically when the server reboots, second, it enables you to start/stop the directory service as the root user without having to switch to the ldap user first.<br /><br />If you installed using the Native Package install, it's easy, since the installation contains a built-in SMF manifest. Just run the following command to activate it:<br /><br /><pre class='postCode'>dsadm enable-service --type SMF /opt/ds_homedir</pre><br /><br />However, if you used the ZIP distribution, it's a little more involved, since you have to create your own SMF manifest and register it manually. Below is the one that I use. You'll need to modify the exec commands to specify the directory of your instance. The timeouts are purposely long, as DS will often need a long time to re-check the database after an ungraceful shutdown (i.e. system crash):<br /><br /><pre class='postCode'><br /><?xml version='1.0'?><br /><!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'><br /><br /><!--<br /> Document : ds_ldaptest2.xml<br /> Created on : 23-OCT-2009<br /> Author : Keith Bucher<br /> Description: The SMF Service Manifest file for Directory Server 6<br />--><br /><br /><service_bundle type="manifest"<br /> name="SUNWldap-directory services"><br /> <service name="application/sun/ds" type="service" version="1"><br /> <dependency name="filesystems" grouping="require_all"<br /> restart_on="none" type="service"><br /> <service_fmri value="svc:/system/filesystem/local:default"><br /> </dependency><br /> <dependency name="network" grouping="require_all"<br /> restart_on="none" type="service"><br /> <service_fmri value="svc:/network/initial:default"><br /> </dependency><br /> <exec_method type="method"<br /> name="start"<br /> exec="/opt/sun/ds6/bin/dsadm start --exec /opt/ds_ldaptest2"<br /> timeout_seconds="600"><br /> <method_context working_directory=":default"><br /> <method_credential user="ldap"<br /> group=":default"<br /> privileges="basic,proc_owner,net_privaddr"/><br /> </method_context><br /> <stability value="Evolving"><br /> </exec_method><br /> <exec_method type="method"<br /> name="stop"<br /> exec="/opt/sun/ds6/bin/dsadm stop --exec /opt/ds_ldaptest2"<br /> timeout_seconds="600"><br /> <method_context working_directory=":default"><br /> <method_credential user="ldap"<br /> group=":default"<br /> privileges="basic,proc_owner,net_privaddr"/><br /> </method_context><br /> <stability value="Evolving"><br /> </exec_method><br /> <instance name="default" enabled="false"><br /> <stability value="Evolving"><br /> <template><br /> <common_name><br /><br /> <loctext lang="C"> Directory Server </loctext><br /> </common_name><br /> <documentation><br /> <doc_link name="Directory Server Enterprise Edition 6 Administration Guide" uri="http://docs.sun.com/doc/819-0995/"<br />/><br /> </documentation><br /> </template><br /> </service><br /></service_bundle><br /><br /><br /></pre><br /><br /><br />Once you've created this file, use the following commands to register it with SMF and enable the service:<br /><br /><pre class='postCode'><br />svccfg import /opt/ds_homedir/config/manifest_file.xml<br />svcadm enable svc:/application/sun/ds:default<br /></pre>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com2tag:blogger.com,1999:blog-645524662879057739.post-40327790995800401702009-03-26T19:15:00.000-07:002010-10-15T15:34:30.234-07:00Configuring gzip/deflate compression on IISAn easy way to improve website performance, especially for low-bandwidth viewers, is to use compression to get the data to the user faster. The text portion of web pages (HTML, Javascript, CSS, etc.) compresses very well (90% or so) so the speedups can be significant. The user's web browser needs to support the compression method, this is done with the Accept-Encoding header that the browser will send to the server if it supports compression, for example, the major browsers support deflate and gzip: "Accept-Encoding: gzip, deflate"<br /><br />Static content can be compressed and then cached, only updating when the original files are changed, so it takes very little CPU usage to implement and it's pretty much a no-brainer to enable. Dynamic content (PHP, ASP.NET, etc.) can also be compressed, but since it is compressed separately for every request, it can take up significant amounts of CPU utilization.<br /><br />Below is a quick guide to enable HTTP compression in IIS:<br /><ul><li>Open up IIS manager and open up the properties for all web sites.</li><li>Under the service tab, select the "Compress static files" and enter the value of the Temporary directory where the cached files will be kept. In addition, you should limit the temporary directory size to 1 GB or so.<br /></li><li>This will enable compression for .txt, .htm and .html files. However, .js and .css files can often be just as large as the HTML files.</li><li>To enable compression for other extensions, you'll need to edit the Metabase. I recommend using <a href="http://support.microsoft.com/kb/840671#8">Metabase Explorer</a> for this. You'll need to edit the W3SVC\Filters\Compression\gzip\HcFileExtensions key and change it from "htm,html,txt" to "htm,html,txt,js,css"</li><li>Now restart IIS to make the changes effective.</li></ul>For more details, see the following <a href="http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/25d2170b-09c0-45fd-8da4-898cf9a7d568.mspx?mfr=true">url</a>. This details how to turn it on for specific websites and other configuration options.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-11010555896056971212009-03-25T17:27:00.000-07:002010-10-15T15:35:05.160-07:00FFMPEG PHP installation on windowsWe recently had a request from a client to install FFMPEG and the associated PHP module on a Windows server that was running Plesk 8.2 with PHP 5.2. This posed a bit of a problem because the FFMPEG project only releases source code releases, relying on others to make binary distributions. For linux distributions, it's usually in the package repository, but for windows, you have to dig around the internet until you can find someone that has compiled it for you, or you can compile it yourself. In addition, they refuse to go out of their way to make their code Windows friendly, which is fine, but it makes the process to compile it kind of complicated, since it doesn't work in Microsoft's C/C++ compiler by default, so you have to configure MinGW to compile it.<br /><br />Finding a windows distribution of the main FFMPEG project was fairly easy. This <a href="http://ffmpeg.arrozcru.org/">site</a> distributes binaries and also provides a very good guide on compiling it for yourself.<br /><br />However, the PHP module is much harder. Especially since you have to find a module that is compatible with your version of PHP. Most of the available instructions point to PHP 4 modules, if you need a PHP 4 module, <a href="http://www.ironlasso.com/install_ffmpeg-php_windows.html">here</a> is one good set of instructions. After much digging, I finally found a PHP 5 module <a href="http://www.myownserver.info/index.php/step3-installing-components/58-convertedvidiscripttutorial.html"><del>here</del></a>. <br /><br />UPDATED: This link is now broken, we're currently hosting the PHP 5 module on our website <a href="http://www.halomede.com/files/ffmpeg-php-win32-all.zip">here</a>.<br /><br />A complete set of instructions for completing this installation is below. My environment was Windows 2003, IIS 6, Plesk 8.2 and PHP 5.2.6:<br /><br /><ul><li>If you plan to use the FFMPEG command-line tool, download the binary package from <a href="http://ffmpeg.arrozcru.org/builds/static/ffmpeg-r16537-gpl-static-win32.tar.bz2">arrozcru.org</a>. Use a tool such as 7-zip to decompress it and expand the tar file where you want to place it.</li><li>Download the zip file that includes the ffmpeg-php extension <a href="http://www.imgoingtoreportyou.com/dir1/WindowsServer/Vidiscript_Complete.zip"><del>here</del></a> and uncompress it. It includes several other software packages, you can ignore them, the files that we are interested in are in the ffmpeg-php-win32-all directory. UPDATED: This link is now broken, we're currently hosting the PHP 5 module on our website <a href="http://www.halomede.com/files/ffmpeg-php-win32-all.zip">here</a>.<br /></li><li> Copy the avcodec-51.dll, avformat-51.dll, avutil-49.dll and pthreadGC2.dll files to the c:\windows\system32 directory.</li><li>Copy the php_ffmpeg.dll file to your PHP extensions directory. For me, this was "C:\Program Files\SWsoft\Plesk\Additional\PleskPHP5\ext"</li><li>Edit your php.ini file and add the following bit. For me this was located at "C:\Program Files\SWsoft\Plesk\Additional\PleskPHP5\php.ini"<br /><br /> extension=php_ffmpeg.dll<br /><br /></li><li>If you are running PHP as an ISAPI filter, you'll probably need to restart IIS. I was running it as a CGI and I didn't need to restart.<br /></li></ul>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com13tag:blogger.com,1999:blog-645524662879057739.post-62018276829468671542009-02-13T18:47:00.000-08:002010-10-15T15:35:24.746-07:00Quick guide to UTF-8 in PHP regular expressionsThere's a lot of different issues that come up with any sort of Unicode support in applications. Mainly because things get a lot more complicated than the standard 8-bit ASCII implementation. UTF-8 is the most common encoding in use for Unicode support. There have been many in-depth documents written about this and I encourage you to read <a href="http://www.regular-expressions.info/unicode.html">them </a>if you have to do anything more than modify a couple of lines of code. To modify regular expressions in PHP for UTF-8 characters, here are a couple of steps that can get you off the ground quickly:<br /><br />* Use the preg functions, the ereg functions do not support Unicode and are going away in PHP 6.<br />* To enable unicode for a regex, add a /u flag. i.e.<br /><br /><span style="font-style: italic;">if(preg_match("/[^[:space:]a-zA-Z0-9{1,}/u", $string)) {</span><br /><br />* Find a good table of UTF-8 characters and their hexadecimal value, like this <a href="http://www.utf8-chartable.de/">one</a>. Use this table to look up the Unicode value, it should be in the format U+<span style="font-style: italic;">NNN</span> where the <span style="font-style: italic;">NNN</span> represents a variable length hexadecimal number. For example the euro symbol, € is U+20AC. To use this in a regular expression, use the \x{<span style="font-style: italic;">NNN</span>} format, i.e. \x{20AC}.<br /><br />* For example, to add € to the allowed characters in the above expression, use the following:<br /><br /><span style="font-style: italic;">if(preg_match("/[^[:space:]a-zA-Z0-9\x{20AC}{1,}/u", $string)) {</span>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-19287126600538112812009-02-04T21:44:00.000-08:002010-10-15T15:35:43.463-07:00Comparing email options for small/medium organizationsOften we'll have clients that need mail setup for their domain as almost an afterthought. The primary use of the domain is to serve a website to a global audience, and the e-mail addresses will primarily be used for customer support or communication among a small team. Usually, it's tacked on to the end of a list of requirements for server configuration, i.e. "oh yeah, and we'll need a mail server too." Generally "mail server" in this context means that they want SMTP, POP, IMAP, a webmail interface and an administrative interface that allows them manage their own user accounts and aliases. Unfortunately, this setup is a little more effort than just doing a "yum install mailserver", so this is generally what we recommend for mail server installation:<br /><br /><span style="font-weight: bold;">Don't do it. </span><br />Use a hosted solution instead, email is a headache to administer and is now standardized enough that it can be offered as a commodity. Standard edition Google Apps for business is free and meets most needs well, there are many other good solutions out there. However, this doesn't work for everyone, due to confidentiality concerns, customization needs, etc.<br /><span style="font-weight: bold;"><br />Zimbra</span><br />If you have the horsepower, run Zimbra. It's free, mostly open source and provides all of the functionality that I mentioned above along with several other features like calendaring. Unfortunately, it requires a lot of resources (especially memory) to run all of its separate components, so it usually needs its own server, which is overkill for a lot of <50 user installations.<br /><br /><span style="font-weight: bold;">Postfix + Dovecot + Squirrelmail + postfixadmin.</span><br />If you don't have the resources to run Zimbra and need to run an in-house mail server, this is probably your best bet for putting together a cost-effective system. Unfortunately, it requires more setup time than the other solutions, but you can easily run it on a low-powered server or in a virtual instance.<br /><br /><span style="font-weight: bold;">Qmail + Courier + Squirrelmail + qmailadmin</span><br />I'm not a big fan of qmail, but it does have a fair number of supporters and I support a couple of different installations, so I'll mention it. Similar to the Postfix bundle, it doesn't require many server resources, but can take a fair bit of setup time. However, qmail is generally more difficult to administer, less well supported, etc, so I would recommend using Postfix as your SMTP server.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-57340741476754813202009-01-29T18:46:00.000-08:002010-10-15T15:36:05.347-07:00PHP file upload best practicesWe've had a couple of different scenarios dealing with PHP file upload issues lately. Out of this, I've come up with some guidelines for using the default PHP file upload mechanism.<br /><br />The first thing to do is to make sure that you are dealing with file upload errors properly. If an error occurs, it will be reported in the $_FILES['examplefile']['error'] field and a list of the error codes are detailed <a href="http://www.php.net/manual/en/features.file-upload.errors.php">here</a>.<br /><br /><span style="font-weight: bold;">PHP Settings</span><br /><br />The most common problem is dealing with large file uploads, both due to the size of the files and the time it takes to upload the files from slower connections. Relevant settings that will need to be adjusted for larger files are:<br /><br /><span>upload_max_filesize </span><br /><span>post_max_size<br /></span><span>memory_limit </span><br /><br />In addition, you'll need to make sure that you have the following settings adjusted to allow enough time for uploads to occur:<br /><br /><span>max_execution_time</span><br /><span>max_input_time</span><br /><br />Another method of limiting the size of file uploads is to have a form input with the name of MAX_FILE_SIZE. PHP will cut off uploads that are larger than the value of this field.<br /><br /><br /><span style="font-weight: bold;">Apache Settings<br /><br /></span>By default, most Apache configurations do not need to be adjusted for file uploads. However, if you're using the LimitRequestBody directive, make sure that it is large enough to accomodate the uploaded files.<br /><br /><span style="font-weight: bold;">IIS Settings</span><br /><br />In addition to the PHP settings for time limits, you will probably need to adjust settings in IIS to make sure that your connections do not time out. The most relevant metabase settings are:<br /><br />ConnectionTimeout (defaults to 120 seconds)<br />CGITimeout (defaults to 500 seconds)Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-2312920910056379952009-01-13T18:22:00.000-08:002009-01-14T08:35:41.389-08:00More flash reverse engineeringThe <a href="http://blog.halomede.com/2009/01/flash-config-files-and-mime-types.html">post</a> that I wrote last week that detailed how to use <a href="http://www.nowrap.de/flare.html">Flare</a> to take a look at the ActionScript for a compiled SWF object. This is useful when you are trying to figure out the details of a flash apps behavior, but what if you actually need to change something about the SWF? Well, the same guy that wrote Flare also wrote a tool call <a href="http://flasm.sourceforge.net/">flasm</a> that will disassemble a flash object to a human readable representation of the Flash byte code. It can also reassemble the byte code to a SWF file. This is not action script and unless you're willing to put in the time to become an expert on the flash virtual machine, it's probably not suitable for any major modifications. However, it's great for doing things like changing hard-coded constants.<br /><br />The basic commands are flasm -d somefile.swf > somefile.flm for generating the assembly file, and flasm -a somefile.flm to generate the SWF file.<br /><br />Here's an example of changing a hard-coded URL in a set of SWF files using flasm. flasm is cross-platform, and is available on windows, OS X and linux, these commands are from a linux system.<br /><br /><span style="font-style: italic;"><br />$ for file in `ls *swf`; do /path/to/flasm -d $file > ${file}.flm; done<br />$ perl -pi -e 's/http:\/\/www\.oldsite\.com\//https:\/\/www\.newsite\.com\//g' *flm<br />$ for file in `ls *flm`; do ~/flasm/flasm -a $file ; done<br /></span>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-90242856872478728652009-01-09T19:33:00.000-08:002010-10-15T15:36:44.295-07:00ApacheCon 2008: Java Monitoring and TroubleshootingI'm finally getting around to watching the presentations from ApacheCon 2008. I plan to blog about them and highlight some of the best things about each session. It costs $99 for access to the <a href="http://streaming.linux-magazin.de/en/archive_apachecon08us.htm">presentations</a>, but is more than worth it if you use Apache, Tomcat, Java or any other Apache project in a production environment.<br /><br />The first one that I watched was Java Monitoring and Troubleshooting, since I've spent a lot of time in the past year chasing down bugs in Java apps. Almost every time I've thought "there's got to be an easier way to do this", now, thanks to Bill Au's presentation, I know some of the easier ways. The materials from his presentation are available <a href="http://people.apache.org/~billa/apacheconus2008/">here</a>. <br /><br />The entire presentation is great, but here were some of the highlights for me:<br /><br />* I've used thread dumps generated by doing "kill -3" on java processes before, but although they contain useful information, I usually spend a lot of time digging for it. He highlights a couple of tools that he uses to analyze thread dumps more efficiently. Basically his methodology is to take three consecutive thread dumps 5 seconds apart. Then he uses the samurai.jar tool in conjunction with his own script overview.pl to analyze the 3 separate files. The overview.pl script gives a good big picture look at how many threads are in each state (runnable, locked, etc.) The samurai tool takes multiple thread dumps and shows how each thread changes state from dump to dump to give a historical perspective. His techniques are especially useful if you're looking for deadlocks or hung threads. Both of these tools are available along with the presentation material.<br /><br />* His demonstration of how to compare the performance of separate garbage collection methods for your application. He starts by getting a log of garbage collection activity for your application for a reasonably long period (an hour or so) when it is under at least moderate use. To do this, use the -Xloggc:<filename> option on your java command line. Get a log file for each GC method that you want to compare. Then download the <a href="http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER">HPjmeter </a>tool released by Hewlett Packard. Then you can open up the logs from HPjmeter and get all sorts of statistics, including how much overhead, the number of full garbage collection events, and average time for each garbage collection. And to make things even better, you can open multiple log files from different GC methods and graph them against each other to get an easy to read picture of the pros and cons of each.<br /><br />* Another neat thing was his demo of <a href="http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html">jhat</a>, the heap analysis tool released by Sun to view the objects in the heap. It's released in Java 6, but can also process heap dumps from 1.4 and 1.5. Once started, it provides a web interface to browse the heap and view objects. The interface make it really easy to track down objects that are taking up more than their fair share of space, but just in case you need more flexibility, there's Object Query Language (OQL). As the acronym suggests, it's a SQL-like interface to the heap. So you can issue statements like this one that finds all finalizable objects and the heap size used by each:<br /><br /><span style="font-style:italic;">select { obj: f.referent, size: sum(map(reachables(f.referent), "sizeof(it)")) }<br /> from java.lang.ref.Finalizer f<br /> where f.referent != null<br /></span><br /><br />Neat stuff! Again, I encourage you to view the entire thing or at least read all of the presentation slides, because all of the information is really good.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-84450926685185898122009-01-06T20:06:00.000-08:002010-10-15T15:37:35.819-07:00Flash config files and MIME typesI ran into this situation for the first time today. A client had purchased a Flash application and couldn't get it to work correctly. The company they bought it from said that the configuration file wasn't getting loaded correctly, but couldn't provide any further guidance. The tech support rep couldn't provide a location of the config file or any details about how it was being loaded. Since the technical support was so lacking and I had no intention to steal their code, I figured that the only thing I could do was take a peek inside the SWF. I used <a href="http://www.nowrap.de/flare.html">Flare</a> to decompile the Actionscript from the SWF and started looking to see how they were loading the configuration. First this tidbit:<br /><br /><span style="font-style:italic;">function loadConfig() {<br /> var v4 = new LoadVars();<br /> v4.onLoad = function (success) {<br /> ....<br /> };<br /><br /> v4.load(_url + '.cfg');<br /> }</span><br /><br />Hmm... So they were using the builtin LoadVars.load() method to retrieve the config file from a URL that ended in ".cfg" Looking a bit further, it turns out that the _url variable was defined as follows:<br /><br /><span style="font-style:italic;">this._url = xmlx.config.requestURL;</span><br /><br />So they're trying to load a file with the same url as the SWF file with a .cfg appended to the end of it. Punching that address into a browser gave a 404 error. After checking to make sure that the permissions were okay and that other files from that directory were accessible, I took a look at the MIME types. This was an IIS server and by default IIS doesn't serve up content without a defined MIME type. As soon as I added ".cfg" as "text/plain" and restarted IIS, everything worked fine.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-31971538302197373172008-12-16T19:33:00.000-08:002010-10-15T15:37:53.851-07:00Sun IDM now supports MySQL in productionFor a long time, Sun has supported use of a MySQL repository for development purposes only with their Identity Manager product. After Sun purchased MySQL, it seemed like only a matter of time until they supported it in production, as Sean O'Neill detailed <a href="http://blogs.sun.com/IdentityCrisis/entry/whats_not_new_in_sun">here</a>.<br /><br />Well finally it looks like there is a low-cost option for IDM repositories. With patch 137621-08, Sun has released support for MySQL in IDM 7.1 and patch 139010-04 does the same for IDM 8.0.<br /><br />We're currently using Oracle and it sure would be nice to move to MySQL. The repository doesn't use any of the advanced functionality that Oracle provides and administering an Oracle database is a lot more work than MySQL. Actually most of the headaches are caused by Oracle's "creative" licensing practices.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-88825160292459845512008-12-10T20:22:00.000-08:002010-10-15T15:38:07.305-07:00#@%#$ user-customizable config files (AKA nss_ldap configuration)I just finished configuring a RHEL system to pull its account information from a central LDAP directory. This shouldn't have been a big deal, I've done it many times before in a lot of different permutations. But this time, TLS certificate verification just would not work no matter what I did. I'd copied the certificate from the LDAP server, put in all of the possible configuration options possible and still nothing. <br /><br />My tests worked with openssl:<br /><span style="font-style:italic;"><br /># openssl s_client -connect hostname:636 -CAfile /etc/openldap/cacerts/hostname.domain.com.cert<br />...<br /><snip><br />...<br /> Verify return code: 0 (ok)<br />---<br />DONE<br /></span><br /><br />but not ldapsearch or getent passwd:<br /><br /><span style="font-style:italic;"># ldapsearch -v -D "uid=user,ou=services,dc=domain,dc=com" -w password -H "ldaps://hostname.domain.com:636" -b "dc=domain,dc=com" -s sub -Z -x "(uid=b*)"<br />ldap_initialize( ldaps://hostname.domain.com:636 )<br />ldap_start_tls: Can't contact LDAP server (-1)<br />ldap_bind: Can't contact LDAP server (-1)<br /></span><br /><br />After scratching my head for a while, fiddling with a bunch of configuration options (which I'll talk more about later), I finally noticed the mention of .ldaprc files in the ldap.conf(5) man page. These are user-customizable config files similar to .bashrc or .netrc that sit in a user's home directory and override the global configuration values. I check the /root/.ldaprc file and bingo!!! There's a reference to a non-existent certificate for the TLS_CERT setting. I comment this out and everything starts working. It turns out that a developer had been experimenting with client certificate authentication to LDAP and had left this setting in after their PoC was done.<br /><br />On a side note, nss_ldap configuration is confusing even without random .ldaprc files getting in your way because you have to deal with both the /etc/ldap.conf and /etc/openldap/ldap.conf files and it is not at all clear what should go in which file. Many of the configuration options overlap, so you will often see duplicate settings in both files. A benefit of all the experimentation that I had to do to solve this problem was that I have a much clearer idea of what needs to be set in which file. For our setup (RHEL 4 client, SSL LDAP server on port 636, no anonymous LDAP access) I configured the following:<br /><br />/etc/ldap.conf:<br /><span style="font-style:italic;"><br /># hostname<br />host hostname.domain.com<br /><br /># search base<br />base dc=domain,dc=com<br /><br /># service user to perform searches<br />binddn uid=user,ou=services,dc=domain,dc=com<br /><br /># service user password<br />bindpw password<br /><br /># port<br />port 636<br /><br /># passwd file configuration<br />nss_base_passwd dc=domain,dc=com?sub<br /><br /># enable ssl<br />ssl on<br /><br /># Verify the certificate<br />tls_checkpeer yes<br /><br /># disable SASL<br />sasl_secprops maxssf=0<br />use_sasl off<br /></span><br /><br />/etc/openldap/ldap.conf: (only one line, but it's vital!)<br /><span style="font-style:italic;"># The location of the trusted certificate (or its CA certificate)<br />TLS_CACERT /etc/openldap/cacerts/watson.uoregon.edu.cert<br /></span>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-38154451165141579792008-12-03T18:19:00.000-08:002010-10-15T15:38:29.147-07:00Zimbra 5 memory usage tweakingWe support several small installations of Zimbra (less than 20 users) along with a couple of larger installations. Everyone loves the features that it provides and the price tag for the ZCS edition. In the past six months we've upgraded several clients from Zimbra 4 to Zimbra 5. In most of the cases, the driving factor was support for Firefox 3 and/or Safari 3.<br /><br />One thing that has come up in the small Zimbra upgrades is a big increase in memory usage with the default settings. The default Zimbra settings are appropriate for installations supporting a couple of hundred users, but are overkill for for small user bases. Below is a set of the steps that we've taken with the small installations:<br /><br />* Reduce the amount of memory that the Java mailbox process and the MySQL server can use. By default these are 30% and 40% respectively. We've been using 25% for both and seen no problems so far.<br /><br />zmlocalconfig -e mailboxd_java_heap_memory_percent=25<br />zmlocalconfig -e mysql_memory_percent=25<br /><br />* Reduce the number of amavisd processes that are started by default. Amavis is a mail content checker that is used by the anti-spam and anti-virus components. By default 10 processes start up, each taking up about 45 MB of memory. We've reduced this to two without seeing any ill effects.<br /><br />Edit $ZIMBRA_HOME/conf/amavisd.conf.in and change this line:<br />$max_servers = 10;<br />to:<br />$max_servers = 2;<br /><br />* If you are really strapped for memory, you can also disable anti-virus, but make sure you have some other method of virus filtering enabled first! Whether it's on a mail filtering service that sits in front of your mail server or on the desktop, you need some sort of protection.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-87718825498855112512008-11-13T19:18:00.000-08:002010-10-15T15:38:45.352-07:00More Plesk tips for troubleshootingI must say that given the complexities of the amount of software packages that control panels (Plesk, cPanel, etc.) have to manage, they do a remarkably good job. Especially since the have to coordinate changes among dozens of different virtual hosts as well. And in Plesk's case, it does it in a cross-platform compatible way as well, working on Windows/IIS as well as Linux/Apache. But it still manages to frustrate me on a weekly basis with a head-scratching problem. This week went over quota with two problems, one of which I described <a href="http://blog.halomede.com/2008/11/plesk-license-auto-renewal-failures.html">here</a>.<br /><br />The second was caused after we upgraded a client's MailEnable installation from 3.14 to 3.6 on their Windows 2003/IIS Plesk server. From their own <a href="http://kb.parallels.com/en/1383">KB article</a>, SWsoft says that you just have to follow the standard MailEnable upgrade instructions and Plesk will automatically detect the new version, so everything will be hunky-dory.<br /><br />Well, after I finished the MailEnable upgrade, things were definitely not hunky-dory. The admin control panel was breaking with regularity, as the application pool repeatedly died and the Plesk event log filled up with errors. After shortening the process recycle time from 1740 to 240 minutes and upping the max processes from 1 to 4, the PleskControlPanel application pool stabilized. However, we were still seeing problems whenever we added or removed a domain. Multiple errors showed up in the Plesk event log similar to the following:<br /><br /><br /><span style="font-style: italic;">Execute websrvmng --start-vhost "--vhost-name=domainname.com" failed: Site domainname.com</span><br /><span style="font-style: italic;">Execute file name: C:\Program Files\SWsoft\Plesk\admin\bin\domainmng.exe</span><br /><br />After digging through some forums and digging through the logs without finding a conclusive cause, I decided to go with my intuition. We'd had problems with MailEnable changing IIS settings before, so the best bet was to re-sync the IIS configuration from the stored Plesk configuration data using the websrvmng utility. Not much documentation exists for the *mng.exe command-line utilities found in the %plesk_bin% directory, but I've found them invaluable for fixing unknown problems such as this. I ran the following commands and after 10 minutes of processing, all was well again (fingers crossed).<br /><br /><span style="font-style: italic;">websrvmng --reconfigure-all</span><br /><span style="font-style: italic;">ftpmng --reconfigure-all</span>Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-50047070002768198852008-11-11T20:22:00.000-08:002010-10-15T15:39:01.501-07:00Plesk License auto-renewal failures solved!We've been battling a strange problem with Plesk licensing for several months. Plesk apparently requires your server to re-register itself with them every couple of months to make sure that its license key is still valid. If the key isn't valid, Plesk enters a grace period after which it will stop working. We've been seeing the following error whenever we try to renew the key:<br /><br /><p><em>Key Update Status:<br />Unable to update Plesk Key. An error occurred while processing your key.</em></p> <p><em>You can try to update it later. The key cannot be upgraded due to the network failure during connection with the Key Authority server. Please check that your Internet connection is configured, you can resolve and access ka.swsoft.com and your firewall enables outgoing connections to TCP port 5224.</em></p><br />Of course, we have no problems connecting to port 5224 on ka.swsoft.com:<br /><br />telnet ka.swsoft.com 5224<br />Trying 64.131.90.38...<br />Connected to ka.swsoft.com.<br />Escape character is '^]'.<br />....<br /><br />The next step is obviously to call SWSoft's support, right? Well when we did that, we found out that since this server is leased from the Planet, they technically own the Plesk license and we'll have to go through their support. This is always a bad sign, when you have separate vendors support people talking to each other, they usually seem to be interested in seeing how they can lay the blame for the problem on the other vendor (and close out their own ticket). True to form, the first two times that our Plesk license was expiring, the Planet installed a temporary key and said they would follow up with SWSoft for a permanent solution. Of course, once the support ticket was closed we never heard back from them.<br /><br />But the third time was the charm. Not only did I get a competent support person, but enough people had complained about the problem that SWSoft had released a <a href="http://kb.parallels.com/en/3969">document </a>on how to fix the problem. It comes down to some corrupt MySQL rows in their license key storage table. Once those are removed, everything works just fine. Of course they never explain why those corrupt rows are there. It appears to be the same two corrupt rows for their example and our server, which makes me think that it is a widespread problem (perhaps caused by an upgrade), but I suppose that I have more pressing issues to deal with now.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-58546944238174235142008-11-07T18:48:00.000-08:002010-10-15T15:39:19.046-07:00PHP ProfilingFor the past couple of days I've been troubleshooting performance problems with a Mediawiki installation on a server we maintain. It's run me through a lot of hoops trying to pinpoint the performance bottleneck. I didn't see anything obvious in the Apache or MySQL logs (although I did learn about the <a href="http://dev.mysql.com/doc/refman/5.0/en/slow-query-log.html">mysqldumpslow</a> command, which makes digging through the slow query logs a lot easier).<br /><br />Next I went to the <a href="http://xdebug.org/">Xdebug </a>PHP extension. I've used it for debugging PHP problems before and I knew that it had a profiling component as well. Installation is a breeze if you have pecl installed:<br /><br />pecl install xdebug<br /><br />Edit php.ini and put in the following to enable profiling:<br /><br />zend_extension=/usr/lib/php/modules/xdebug.so<br />[xdebug]<br />xdebug.default_enable=Off<br />xdebug.profiler_enable=On<br /><br />I disabled the default stack traces since it was a production environment. I only left the profiler enabled for enough time for me to run my tests, since it generates a lot of information for each process. Of course, I guess I could have just enabled it for the Mediawiki directory if that had been a big issue, but it was a pretty low-traffic time, so I only had 15 files to sort through from my testing period. I copied over the /tmp/cachegrind.out.* files to my desktop and fired up <a href="http://xdebug.org/docs/profiler">KCachegrind to analyze the output</a>.<br /><br />It was much more enjoyable to use the color-coded GUI than looking through a text file. After finding the correct output file, I traced the bottleneck down to the MessageCache->lock() method. This Mediawiki installation uses memcached to cache db objects and it was just spinning there for insane amounts of time trying to lock objects in memcached.<br /><br />Next I went to the Mediawiki <a href="http://www.mediawiki.org/wiki/How_to_debug_MediaWiki#In-depth_debugging">debug log</a>. As a side note, Mediawiki has its own <a href="http://www.mediawiki.org/wiki/How_to_debug_MediaWiki#Profiling">profiling setup </a>that you can use as well, although I prefer the GUI that Xdebug uses. I saw messages like this:<br /><br />MessageCache::load(): cache is empty<br /><br />Then 10 seconds or so later:<br /><br />MessageCache::load(): unable to load cache, disabled<br /><br />After a couple of false starts, including playing around with the <a href="http://us.php.net/apc">APC cache</a> as a replacement, it turns out that the memcached process Mediawiki was trying to use didn't exist. Apparently the startup commands had been put in /etc/rc.local and hadn't included the full path, so after the last reboot of the system, memcached hadn't restarted. A proper <a href="http://www.dev411.com/wiki/Memcached_startup_files_for_Red_Hat_%28RHEL%29">init script</a> and a couple of chkconfig commands later, all was well.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0tag:blogger.com,1999:blog-645524662879057739.post-2460824163382125692008-11-05T21:12:00.000-08:002010-10-15T15:39:33.372-07:00MySQL WorkbenchI've used DBDesigner in the past for MySQL data modeling, but that was discontinued a while back. It looks like Workbench might be my new tool for MySQL schema designing and visualization. I needed to do some schema comparisons from existing DBs today to write an upgrade script for a PHP/MySQL web application and Workbench wound up having just the features that I needed.<br /><br />The steps that I needed to do to get the upgrade script were covered by this <a href="http://simpcode.blogspot.com/2008/06/using-mysql-workbench-to-generate.html">post</a>. I'll keep evaluating it to see if I need the commercial version or not, but it's so cheap ($99), that I'll probably wind up buying it.Keith Bucherhttp://www.blogger.com/profile/03577005731261749597noreply@blogger.com0