• For 1.9.3, I ran "bundle install", but failed in compiling mongrel-1.1.5. mongrel-1.1.5 is not compatible with Ruby 1.9.3. Using mongrel-1.2.0pre2, it will be ok. You need to rerun "./configure" if you change the ruby version in RVM. And when you install, you have to pass your ANT_HOME and rvm_path if root doesn't have them.

  vi thrift-0.8.0/lib/rb/thrift.gemspec
  
    s.add_development_dependency "mongrel", "1.2.0pre2"
  
  cd thrift-0.8.0/lib/rb
  bundle install
  cd ../../..
  ./configure
  make
  sudo ANT_HOME=$ANT_HOME rvm_path=$rvm_path bash -c "source /home/bewang/.rvm/scripts/rvm && rvm use 1.9.3 && make install"
  

• For 1.8.7, I ran "bundle install" without any problem, but the compilation failed sometimes with this rspec error.

  Spec::Mocks::MockExpectationError in 'Thrift::UNIXSocket should raise an error when read times out'
   expected :select with (any args) once, but received it twice
  /home/bewang/temp/thrift-0.8.0/lib/rb/spec/socket_spec_shared.rb:94:
  

sudo yum install automake libtool flex bison pkgconfig gcc-c++ boost-devel libevent-devel zlib-devel python-devel ruby-devel
tar zxvf thrift-0.8.0.tar.gz
./configure --without-ruby
make
sudo ANT_HOME=$ANT_HOME make install 

• Hive server 2 support LDAP and Kerberos, but only supports Simple bind. Unfortunately our LDAP server only supports SASL. If you get an error saying "Error validating the login", you may have the same issue I had. Just try to use ldapsearch in the command line to verify if you can access the ladp server. And you'd better take a look of /etc/ldap.conf if you use CentOS.

ldapsearch -Z -x "uid=bewang"

java.sql.SQLException: Method not supported
        at org.apache.hive.jdbc.HiveDatabaseMetaData.supportsMixedCaseIdentifiers(HiveDatabaseMetaData.java:922)

• There are two networks: corp.mycompany.com and lab.mycompany.com.
• lab trusts corp, but corp doesn't trust lab

1. Browser sends GET /hello_spnego.jsp
2. Server returns 401 Unauthorized with Negotiate

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1\r\n
WWW-Authenticate: Negotiate\r\n
WWW-Authenticate: Basic realm="LAB.MYCOMPANY.COM"\r\n

3. Client sends KRB5 TGS-REQ
4. Client receives KRB5 KRB Error: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN

Kerberos KRB-ERROR
  Pvno: 5
  MSG Type: KRB-ERROR(30)
  stime: 2012-10-10 23:04:48 (UTC)
  susec: 394362
  error_code: KRB5KDC_ERR_S_PRINCIPAL_UNKNOWN
  Realm: CORP.MYCOMPANY.COM
  Server Name (Service and Instance): HTTP/tomcat.lab.mycompany.com

5. Browser sends GET /hello_spnego.jsp HTTP/1.1, NTLMSSP_NEGOTIATE

spnego-client {
  com.sun.security.auth.module.Krb5LoginModule required;
};

spnego-server {
  com.sun.security.auth.module.Krb5LoginModule
    required
    useKeyTab=true
    keyTab="conf/appserver.keytab"
    principal="serviceaccount@CORP.MYCOMPANY.COM"
    storeKey=true
    isInitiator=false;
};

[libdefaults]
  default_realm = LAB.MYCOMPANY.COM
  default_tgs_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des3-hmac-sha1
  default_tkt_enctypes = arcfour-hmac-md5 des-cbc-crc des-cbc-md5 des3-hmac-sha1
  clockskew = 300

[realms]
  LAB.MYCOMPANY.COM = {
    kdc = kdc1.lab.mycompany.com
    kdc = kdc2.lab.mycompany.com
    default_domain = lab.mycompany.com
  }                         

[default_realm]             
  lab.mycompany.com = LAB.MYCOMPANY.COM
  .lab.mycompany.com = LAB.MYCOMPANY.COM

1. If I don't quote the principal like this principal=serviceaccount@CORP.MYCOMPANY.COM, I will get the configuration error. And the message is misleading because line 9 is keyTab.

Caused by: java.io.IOException: Configuration Error:
        Line 9: expected [option key], found [null]

2. When you use ktab, the first thing you need to know is only windows version has this tool, while Linux RPM from oracle doesn't have it.
3. You should use the service account in corp network, not lab, to generate the keytable file like this:

ktab -a serviceaccount@CORP.MYCOMPANY.COM -k appserver.keytab

4. Make sure your
5. I also encountered this error: KrbException: Specified version of key is not available (44). It turns out that the keytab file I generated with kvno=1 and the expected is 2. You can use wireshark to catch the packet for KRB5 TGT-REP, and it will tell you what kvno is expected.

Ticket
  Tkt-vno: 5
  Realm: LAB.MYCOMPANY.COM
  Server Name ....
  enc-part rc5-hmac
    Encryption type: ...
    Kvno: 2 *** Here it is
    enc-part: ...

6. You have to run ktab command multiple times to achieve the correct kvno just like this page http://dmdaa.wordpress.com/2010/05/08/how-to-get-needed-kvno-for-keytab-file-created-by-java-ktab-utility/. Use can just use ktab -l to find the kvno:

ktab -l -k appserver.keytab

7. Which version of JDK seems not important. A keytab file generated by JDK 7 worked in JDK 1.6.0_32.
8. I also got this Checksum error if I used my lab service account (serviceaccount@lab.mycompany.com) in pre-auth fields or keytab.

SEVERE: Servlet.service() for servlet [jsp] in context with path [] threw exception [GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] with root cause
java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.ArcFourCrypto.decrypt(ArcFourCrypto.java:388)
        at sun.security.krb5.internal.crypto.ArcFourHmac.decrypt(ArcFourHmac.java:74)

1. db2ls can show the current installation

[bewang@logs]$ db2ls
Install Path                       Level   Fix Pack   Special Install Number   Install Date                  Installer UID 
---------------------------------------------------------------------------------------------------------------------
/opt/ibm/db2/V9.7                 9.7.0.2        2                            Tue Oct  4 17:08:48 2011 PDT             0 

2. and the installed components

[bewang@logs]$ db2ls -q -b /opt/ibm/db2/V9.7

Install Path : /opt/ibm/db2/V9.7

Feature Response File ID             Level   Fix Pack   Feature Description  
---------------------------------------------------------------------------------------------------------------------
BASE_CLIENT                         9.7.0.2          2   Base client support 
JAVA_SUPPORT                        9.7.0.2          2   Java support 
LDAP_EXPLOITATION                   9.7.0.2          2   DB2 LDAP support 

3. db2idrop to remove, then db2_deinstall

I thought it should not be too hard to import users from Hue 1.2.0 into Hue 2 (CDH 4.1) because this page doesn't mention special steps: https://ccp.cloudera.com/display/CDH4DOC/Hue+Installation#HueInstallation-UpgradingHuefromCDH3toCDH4.

But I was wrong. After importing auth_user, auth_group, and auth_user_groups and successfully log on using my old username and password, I got "Server Error (500)" and I couldnot find any error message in the log files.

It turns out that you have to create records in useradmin_grouppermission and useradmin_userprofile tables for each user in Hue 2(CDH4.1). Here are the queries:

mysql> insert into useradmin_grouppermission(hue_permission_id,group_id) select hp.id, g.id from useradmin_huepermission hp inner join auth_group g;
mysql> insert into useradmin_userprofile (user_id, creation_method, home_directory) select u.id, 'HUE', concat('/user/', u.username) from auth_user u;

• drop database hue
• create database hue
• build/env/bin/hue syncdb
• answer no when you are asked if creating a superuser

• 
  You just installed Django's auth system, which means you don't have any superusers defined.
  Would you like to create one now? (yes/no): no
  

• mysqldump -uxxx -pxxxx -h old_db --compact --no-create-info --disable-keys hue auth_group auth_user auth_user_groups auth_user_user_permissions > hue_user.sql
• mysql -uyyy -pyyyy -h new_db -D hue
• run the above insert queries

• Don't use utf8 as default. Change /etc/my.cnf to use latin1. For latin1, varchar can hold 64K. After syncdb, just modify the tables charset to use utf8 if you still want utf8.
• Edit the migration to replace all 32678 to 12678 before running syncdb. It doesn't matter to what value you change because the developers already have the fix in migration 0003_xxx of the same directory and the table will be fixed anyway. Don't forget to delete 0002_.pyc in the directory.

• Fingerprint Issue in Jenkins: delete the dir d:\.jenkins\fingerprints

Waiting for Jenkins to finish collecting data
ERROR: Asynchronous execution failure
java.util.concurrent.ExecutionException: hudson.util.IOException2: Unable to read D:\.jenkins\fingerprints\42\e9\40d5d2d822f4dc04c65053e630ab.xml
   ...
Caused by: hudson.util.IOException2: Unable to read D:\.jenkins\fingerprints\42\e9\40d5d2d822f4dc04c65053e630ab.xml
   ...
Caused by: com.thoughtworks.xstream.io.StreamException:  : only whitespace content allowed before start tag and not \u0 (position: START_DOCUMENT seen \u0... @1:1)
   ...
Caused by: org.xmlpull.v1.XmlPullParserException: only whitespace content allowed before start tag and not \u0 (position: START_DOCUMENT seen \u0... @1:1)

    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
    http://maven.apache.org/xsd/settings-1.0.0.xsd">
/home/bewang/temp/maven/.m2/repository






nexus

true



my-repo
http://nexus-server:8080/nexus/content/groups/public/url>

true


true







nexus-public
*,!eclipselink-repo
http://nexus-server:8080/nexus/content/groups/pu
blic


nexus-eclipselink
eclipselink-repo
http://nexus-server:8080/nexus/content/repositor
ies/eclipselink-maven-mirror

My company has a lengthy process if we need root permission to deploy something into a Linux server, 3-4 days for approval. Sound impossible? Unfortunately it is true. And I don't want to run my puppet modules under root permission because they have to be managed by IT team and it could take even longer to deploy and maintain those modules. My goal is to avoid any root permission as possible as I can.

Actually it is possible that run multiple puppet agents in one machine. One agent is managed by IT, who handles the stuffs I don't care, such as NTP. One agent can run under my control with a non-root user with some sudo privileges, like run /sbin/service, etc. And I can manage a puppet master and config my agents to connect to this master to do the deployment.

It is actually pretty simple, even without any configuration, just provide confdir, vardir and server in the command line.

puppet master --confdir=/home/bewang/.puppet --vardir=/home/bewang/.puppet/var --modulepath=/home/bewang/modules --pluginsync --no-deamonize --debug --verbose
puppet agent --confdir=/home/bewang/.puppet --vardir=/home/bewang/.puppet/var server=my-pp-master --test

puppet cert --confdir=/home/bewang/.puppet --vardir=/home/bewang/.puppet/var list
puppet cert --confdir=/home/bewang/.puppet --vardir=/home/bewang/.puppet/var sign my-pp-agent-host-name

service { "my-service":
  ensure => running,
  start => "sudo /sbin/service my-service start",
}

Deployment of a Java application can be done in a different ways. We can do

• Build one jar with all exploded dependent jars and your application. You don't have to worry about your classpath.
• Build a tarball. It is much easier using Maven assembly and Maven AppAssembly plugin to build a tarball/zip to include all dependent jars and the wrapper scripts for running application or service. AppAssembly generate the classpath in the wrapper scripts.
• Build a war and deploy it to the container.

All above methods have the same issue: you will repeatedly deploy the same jars again and again. One of my project uses a lot of third-party libraries, Spring, Drools, Quartz, EclipseLink and Apache CXF. The whole zip file is about 67MB with all transitive dependent jars, but my application is actually less than 1MB. Every time I have to build and store such large files, and include all dependent jars without any change. Even the hard drive is much cheaper today, doing like this is still not a good practice.

It is easy to deploy my application jar and replace the old jar only each time if I don't update any dependency. Otherwise, it is still inconvenient because you have to change multiple jars.

The ideal way is to deploy the application jar and dependent jars in a directory other than the application directory on the target server, and build the classpath each time when a new version or dependency change is deployed. There are the following advantages:

• Dependent jars only needs to be deployed once.
• Different applications can share the same dependencies.
• Multiple versions exist on the target server. It is much easier to rollback to the old version.
• You save a lot of space and network traffic

You probably get the answer: Maven. Using maven, maven dependency plugin, and maven local repository, you can simply and easily implement such a system.

I have a working puppet module can do the followings:

• setup maven from tarball;
• setup maven settings.xml, local repository, and you maven repository on the intranet;
• resolve the dependencies of a maven artifact: download jars into the local repository;
• setup symlinks of dependent jars into a directory so that you don't have different copies for different applications;
• generate a file having the classpath.

I will publish it to github once I get the time.

You can find similar puppet modules in github, like puppet-nexus and puppet-maven. They just copy the specified jars to a directory.

Hive 0.9.0 supports remote debug.Run the following command, and Hive will suspend and listening on 8000.

hive --debug

[bewang@myserver ~]$ hive --debug
ERROR: Cannot load this JVM TI agent twice, check your java command line for duplicate jdwp options.
Error occurred during initialization of VM
agent library failed to init: jdwp

By setting xtrace, I found "-XX:+UseParallelGC -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y" is actually add twice. In this commit 54abc308164314a6fae0ef0b2f2241a6d4d9f058, HADOOP_CLIENT_OPTS is appended to HADOOP_OPTS, unfortunately this is done in $HADOOP_HOME/bin/hadoop".

--- a/bin/hive
+++ b/bin/hive
@@ -216,6 +216,7 @@ if [ "$DEBUG" ]; then
   else
     get_debug_params "$DEBUG"
     export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS $HIVE_MAIN_CLIENT_DEBUG_OPTS"
+    export HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
   fi
 fi

Removing this line will fix the issue.

Just found this article Batch fetching - optimizing object graph loading

We have some tables with 15K ~ 20K partitions. If I run a query scanning a lot of partitions, Hive could use more than 10 minutes to commit the mapred job.

The problem is caused by ObjectStore.getPartitionsByNames when Hive semantic analyzer tries to prune partitions. This method sends a lot of queries to our MySQL database to retrieve ALL information about partitions. Because MPartition and MStroageDescriptor are converted into Partition and StorageDescriptor, every field will be accessed during conversion, in other words, even the fields has nothing to do with partition pruning, such as BucketCols. In our case, 10 queries for each partition will be sent to the database and each query may take 40ms.

This is known ORM 1+N problem. But it is really bad user experience.

Actually we assembly Partition objects manually, it would only need about 10 queries for a group of partitions (default size is 300). In our environment, it only needs 40 seconds for 30K partitions: 30K / 300 * 10 * 40.

I tried to this way:

1. Fetch MPartition with fetch group and fetch_size_greedy, so one query can get MPartition's primary fields and MStorageDescriptor cached.
2. Get all descriptors into a list "msds", run another query to get MStorageDescriptor with filter like "msds.contains(this)", all cached descriptors will be refreshed in one query instead of n queries.

This works well for 1-1 relations, but not on 1-N relation like MPartition.values. I didn't find a way to populate those fields in just one query.

Because JDO mapping doesn't work well in the conversion (MPartition - Partition), I'm wondering if it is worth doing like this:

1. Query each table in SQL directly PARTITIONS, SDS, etcs.
2. Assembly Partition objects

This is a hack and the code will be really bad. But I didn't find JDO support "FETCH JOIN" or "Batch fetch".

Recently I wrote a post for Bad performance of Hive meta store for tables with large number of partitions. I did tests in our environment. Here is what I found:

• Don't configure a hive client to access remote MySQL database directly as follows. The performance is really bad, especially when you query a table with a large number of partitions.

javax.jdo.option.ConnectionURL
jdbc:mysql://mysql_server/hive_meta


javax.jdo.option.ConnectionDriverName
com.mysql.jdbc.Driver


javax.jdo.option.ConnectionUserName
hive_user


javax.jdo.option.ConnectionPassword
password

hive --service metastore
# If use CDH
yum install hive-metastore
/sbin/service hive-metastore start

hive.metastore.uris
thrift://mysql_server:9083

ERROR conf.HiveConf: Found both hive.metastore.uris and javax.jdo.option.ConnectionURL Recommended to have exactly one of those config keyin configuration

1. Get a list of partition names using db.getPartitionNames()
2. Then call db.getPartitionsByName(List<Strin> partNames). If the list is too large, it will load in multiple times, 300 for each load by default. The JDO calls like this
   • For one MPartition object.
   • Send 1 query to retrieve MPartition basic fields.
   • Send 1 query to retrieve MStorageDescriptors
   • Send 1 query to retrieve data from PART_PARAMS.
   • Send 1 query to retrieve data from PARTITION_KEY_VALS.
   • ...
   • Totally 10 queries for one MPartition. Because MPartition will be converted into Partition before send by, all fields will be populated
3. If one query takes 40ms in my environment. And you can calculate how long does it take for thousands partitions.
4. Using remote Hive metastore service, all those queries happens locally, it won't take that long for each query, so you can get performance improved significantly. But there are still a lot of queries.

I also wrote ObjectStore using EclipseLink JPA with @BatchFetch. Here is the test result, it will at least 6 times faster than remote metastore service. It will be even faster.

According to puppet reference include and require:

• Both include and require are functions;
• Both include and require will "Evaluate one or more classes";
• Both include and require cannot handle parametered classes
• require is a superset of include, because it "adds the required class as a dependency";
• require could cause "nasty dependency cycle";
• require is "largely unnecessary"; See puppet language guide.
  Puppet also has a require function, which can be used inside class definitions and which does implicitly declare a class, in the same way that the include function does. This function doesn’t play well with parameterized classes. The require function is largely unnecessary, as class-level dependencies can be managed in other ways.
• We can include a class multiple times, but cannot declare a class multiple times.

  
  class inner {
    notice("I'm inner")
  
    file {"/tmp/abc":
      ensure => directory
    }
  }
  
  class outer_a {
    # include inner
    class { "inner": }
  
    notice("I'm outer_a")
  }
  
  class outer_b {
    # include inner
    class { "inner": }
  
    notice("I'm outer_b")
  }
  
  include outer_a
  include outer_b
  

  
  Duplicate declaration: Class[Inner] is already declared in file /home/bewang/temp/puppet/require.pp at line 11; cannot redeclare at /home/bewang/temp/puppet/require.pp:18 on node pmaster.puppet-test.com
  

• You can safely include a class, first two examples pass, but you cannot declare class inner after outer_a or outer_b like the third one:

  
  class inner {
  }
  
  class outer_a {
    include inner
  }
  
  class outer_b {
    include inner
  }
  
  class { "inner": }
  include outer_a
  include outer_b
  

  
  class { "inner": }
  class { "outer_a": }
  class { "outer_b": }
  

  
  include outer_a
  include outer_b
  class { "inner": } # Duplicate redeclaration error
  

#!/usr/bin/env ruby
require 'rubygems'

require 'pty'
require 'expect'
require 'io/console'

hosts_file = ARGV[0] || "hosts"

print "Password:"
password = $stdin.noecho(&:gets)
password.chomp!
puts

$expect_verbose = true
File.open(hosts_file).each do |host|
  host.chomp!
  print "Copying id to #{host} ... "
  begin
    PTY.spawn("ssh-copy-id #{host}") do |cp_out, cp_in, pid|
      begin
        pattern = /#{host}'s password:/
        cp_out.expect(pattern, 10) do |m|
          cp_in.printf("#{password}\n")
        end
        cp_out.readlines
      rescue Errno::EIO
      ensure
        Process.wait(pid)
      end
    end
  rescue PTY::ChildExited => e
    puts "Exited: #{e.status}"
  end
  status = $?
    if status == 0 
      puts "Done!"
  else
    puts "Failed with exit code #{status}!"
  end
end

select * from my_table where date_column = '2013-01-11';

ssh myserver 'hive -e "select * from my_table where date_column = '"'2013-01-11'"';"'

1. ssh
2. myserver
3. hive -e "select * from my_table where date_column = 
4. '2013-01-01'
5. ;"

hive -e "select * from my_table where date_column = '2013-01-01';"

Requirements: 

1. Include scala-library.jar in the zip file because the machine may not have scala installed. 
2. Include scripts and configure files into the zip just like Maven assembly. 
3. Include the jar in the zip file. 

 Lessons:

settings // list settings keys
tasks // list all tasks
show clean-files // check value of settings keys
inspect clean-files // check more information than show

svn diff somefile | vim -

• Clone a project. You can run the following commands if the project follows the standard layout with trunk/tags/branches.

  root
    |- ProjectA
    |   |- branches
    |   |- tags
    |   \- trunk
    |- OtherProject
  

• Download trunk, tags and branches. You will find the same directories just like SVN repository.

  git svn clone -r XXXX https://subversion.mycompany.com/ProjectA
  

• init a new project. This will create dir projectA and initializ it as a git repository.

  git svn init https://subversion.mycompany.com/projectA projectA
  

• Download trunk only. The directory have the files under trunk only.

  git svn clone -s -r XXXX https://subversion.mycompany.com/ProjectA

• Provide the revision number using "-r xxxx", otherwise have to wait for a long time until git-svn finishes scanning all revisions.
• Use the revision of the project (96365) instead of the latest revision (96381). Otherwise, you will get the working directory empty. You can find the revision number in viewvc like this:

  Index of /ProjectA
  
  Files shown: 0
  Directory revision: 96365 (of 96381)
  Sticky Revision:    
  

• Extract GZIP compressed TAR file

  tar xzvf file.tar.gz

• Extract BZ2 compressed TAR file

  tar xvjpf file.tar.gz