How to resolve spark-cassandra-connector's Guava version conflict in spark-shell
In my blog How to resolve spark-cassandra-connector Guava version conflicts in Yarn cluster mode, I explained how to resolve Guava version issue in Yarn cluster mode. This blog covers how to do it in...
View ArticleSpark Cassandra Connector and DataFrame
When you write a DataFrame to a cassandra table, be careful to use SaveMode.Overwrite. In spark-cassandra-connector-1.6.0-M2, TRUNCATE $keyspace.$table will be called. See the code in...
View ArticleHow to make @timestamp using GMT when using Fluentd, Elasticsearch and Kibana?
My log is a JSON one-liner output by a Node.js application, there is a field called “time” which is GMT time. { "req": {}, "time":"2016-05-12T19:18:38.123Z"}I want to keep the timestamp in GMT in...
View ArticleHow to create .epub and .mobi version of Gradle User Guide?
Gradle User Guide is written using docbook, and gradle build already have single HTML and pdf built. But I really want to load it into my kindle. Because docbook supports converting docbook to epub and...
View ArticleBring back google-chrome after upgrading to CentOS 6.8 and Chrome 51.
I don’t know which one is root cause: upgrading to CenOS 6.8 or Chrome 51. I used install_chrome.sh on http://chrome.richardlloyd.org.uk/ to install Google chrome on my CentOS 6 VirtualBox VM. It...
View ArticleMake Spark read Teradata directly.
Spark SQL supports read JDBC resource, but the paragraph in the latest 2.0 document is not really helpful:The JDBC driver class must be visible to the primordial class loader on the client session and...
View ArticleRecord Request and Reponse of HTTP Samplers in JMeter
Add “Simple Data Writer”Give the file name like “result.xml”Click Configure button.Check the radio buttons for Save As XMLSave URLSave Response Data (XML)Start the jmeter testCheck the XML fileIf you...
View ArticleGradle show dependencies of the specified configuration
I record the method here in case I forget how to show the dependencies for ONLY one configuration like runtime again.Show dependencies for one configuration ONLY: ./gradlew dependencies --configuration...
View ArticleDon't set limit nofile to unlimited
After I set nofile to unlimited in /etc/security/limits.d/90-nproc.conf like this,* hard nproc unlimited* hard nofile unlimited* soft nproc unlimited* soft nofile unlimitedI could not boot into Gnome...
View ArticleVNC Viewer
I recently built a desktop for development in my company’s CORP network. By using VNC, I can access the same Gnome session even from home. Just simply connect to VPN, and start a vnc viewer, I can...
View ArticleBuild Git RPM on CentOS 7
I want to use core.hooksPath which supports since 2.9, but the default Git version of CentOS 7.3 is still 1.8.3:$ yum list gitAvailable Packagesgit.x86_64 1.8.3.1-6.el7_2.1 baseI have to compile from...
View ArticleInstall Docker Toolbox for Windows Automatically
Docker Toolbox for Windows have command line arguments which allows you to install it without user’s involvement.You can run this command in Windows Command Prompt to get those arguments:>...
View ArticleInstall Cygwin in a script
I built an Intellij docker image based on CentOS 7. To allow my colleagues to use it on Windows, I need to help them to setup a X window system. This post describe how to setup Cygwin/X using a script...
View ArticleScreen in docker with error "Must be connected to a terminal"
I have a running docker container with command /bin/bash --login. When I run the command$ docker exec -it devsh /bin/bash --loginI can access the container’s Bash, but when I run screen, I got this...
View ArticleGnome Terminal Profile Export
I usually run screen on a remote server in PROD environment. Whenever I want to access the server, I SSH to that host, and run screen -D -R to resume the session I leave before. Using screen, you don’t...
View ArticleMac OS SSH Kerberose Single-Sign-On
To use Kerberos Single-Sign-On on a MacBookPro, you can do as follows:Make a copy of /etc/krb5.conf from a server in your company network.Save it into /etc/krb5.confon your MacBookPro. You probably...
View ArticleSpark SQL AnalysisException due to data type mismatch
If you run the following code, you will encounter AnalysisException with data type mismatch.spark.sql("create table test_table(id int, data struct<name: string, age:int>)")spark.sql("insert...
View ArticleVNC server won't start on my CentOS 7
I had to ask my boss to power down and power up the desktop in my office this morning. After the desktop came back, I could not access the vncserver because it failed to start. And I got this error...
View ArticleAllow the outside world access your server in a container
I run docker containers of Nexus and httpd on a CentOS 7 host. I added a nexus.service to start the containers using docker-compose. I could access the nexus server from any machine after I started the...
View Article"No such a file or directory" when setting NETGEAR router
I got this error when I was setting up a new NETGEAR X6 router after setting up the new password. I restored the factory settings multiple times, but it either showed "No such a file or directory" for...
View ArticleHow to make VNC sharp on Retina display.
Most of time, I work on a Linux box running CentOS 7 through VNC. The largest VNC screen size is 1920x1080 (16:9). It is ok when you maximize the RealVNC viewer window on an external monitor with that...
View ArticleMake Intellij IDEA on Linux with sharp fonts as Macbook Pro
IntelliJ IDEA supports HiDPI on Windows and MacOS. The fonts look so beautiful on a Macbook Pro Retina Display. Unfortunately IDEA doesn’t support HiDPI on Linux. Looks at the code...
View ArticleRead HAR to Spark DataFrame
When I use spark-streaming to pull JSON events from a Kafka topic and persist the data into HDFS, I have to handle a lot of tiny files because the volume of the Kafka topic is pretty small. Too many...
View ArticleAutomic Application Manager V9 Client: Run with Java 9
My team uses Automic AM v9.1.0 (v9.1.0_28363_28431), which is configured with Java 6 and Java 7 by default. It also works in Java 8. However, if you run it with Java 9, you will encounter network error...
View ArticleWSL mv permission denied
If you get "Permission Denied" when you `mv` a file, check if you open the parent directory of the file in Windows Explorer. If so, close Windows Explorer or change it to another directory, and try...
View Article
