About me‎ > ‎Profiles‎ > ‎

Skillmatrix

NOTE: I don't profess mastery but I'm adept with most of the listed, and I have an aptitude towards learning what I need to ;-D (red+bold > bold > normal text)


Network / Infrastructure Architecture

cat > LOOK.OUT <<'EOM'
Look at sample >>>Architectures<<< I work with on a daily basis.
In real life they are more complicated ;-)
EOM

- AWS VPC (network virtualization)
Connectivity model: IPsec VPN, AWS Direct Connect, VPC Peering, OpenVPN (least preferred for site to site).

- On premise (e.g. XenServer networking)
  • Segregating different types of network traffic (management, VM and IP-based storage traffic)
  • Open vSwitch as network backend (with OpenFlow control and optional DVSC)
  • VLANs
  • NIC bonding
    basic OVS modes - active-active (balance-slb) and active-passive
    LACP bond with load balancing based on IP and port of source and destination, or based on Source MAC addresses
  • multipathing (Linux DM-Multipath - does NOT work for NFS - which needs NIC bonding for better throughput)
  • Jumbo Frames (MTU=9000, NO! Not that simple, very tricky)
  • High Availability
  • Failover (network, storage, running VMs, hosts, etc.)
  • Disaster Recovery design

Web Application Architecture


- Load balancing
- Web Server
- Caching
- Middleware
- RDBMS

Linux (click to see more)

A Linux Ninja who started his journey with Mandrake Linux 8.1 in 2001, with deep knowledge and and rock solid skills, you can count on me in the following areas

  • System Administration (Arch Linux, Ubuntu, Debian, Fedora, CentOS, Oracle Linux, RHEL, openSUSE, SLES)
  • Storage / Volume Management
    device mapper, md/mdadm
    LVM
    Btrfs (not recommended for production)
    ZFS (OpenZFS / ZFS on Linux ---> see Netflix use case)
    iSCSI (tgt/STGT, Linux-IO Target, SCST, open-iscsi/iscsiadm CLI utility)
    Open FCoE
    Fibre Channel (HBA) SAN
    Multipathing (DM Multipath)
  • Ceph (object storage and file system)
  • Troubleshooting (strace/ltrace, kernel crash dump analysis using crash, core dump analysis using gdb, etc...)
  • Performance (FlameGraph, perf-tools, bcc)
  • Tuning (kernel, file system, networking etc...)
  • Monitoring
  • Security Best Practice (very high standard for security and privacy)
  • High Availability (Ksplice, kpatch, DRBD)
  • Automation & Configuration Management (Chef, Ansible)
  • Virtualization in the kernel mainline (Xen, KVM, LXC/LXD)

Linux Kernel

Extensive knowledge of Linux kernel, have been closely following kernel development since kernel 2.6.32, many thanks to Linux Kernel Newbies.

Specialized in Kernel Crash Dump analysis, capable of tracing back to kernel source code and do RCA (love Linux Cross Reference powered by LXR!).

Real world experience in Live (Dynamic) Kernel Patching tools
  • Ksplice uptrack (Acquired by Oracle, free for Ubuntu and Fedora) - zero-downtime (rebootless) kernel patching
    Since July 2011 (check my tweets back then).
  • kGraft (by SUSE, main developer - Jiri Slaby) - source code -> kGraft 
  • kpatch (by Red Hat) -> kpatch@GitHub
    NOTE: Happy to be a casual kpatch contributor, mainly worked on adding distribution support (Ubuntu 14.04 LTS, Oracle Linux 7 and CentOS 7 so far) and improved its documentation. Allegedly the 1st person to get kpatch running on Ubuntu ;-D

Package Managers

  • apt / dpkg
  • pacman / yaourt
  • dnf (yum for Enterprise Linux as of EL7) / rpm
  • zypper / rpm

File Systems

  • ext{2,3,4}
  • Btrfs (early adopterr)
  • ZFS (on Solaris 10/11.1 and FreeNAS ZFS v5.0 Storage Pool v28, ZFS on Linux now)
  • XFS (xfsprogs)
  • LVM (black belt)
  • eCryptfs (Enterprise cryptographic filesystem for Linux, filesystem level encryption)
  • dm-crypt / LUKS (disk encryption subsystem in Linux Kernel, part of the device mapper infrastructure, used by Android full disk encryption)
  • FAT16/FAT32/exFAT
  • NFS {v3,v4}
  • SMB / CIFS
  • GlusterFS

Data Recovery

  • testdisk & photorec
    Powerful data recovery software, opensource. The best in breed I have ever used!
  • DiskGenius
    As its name indicates. It does a decent job.

Networking

Linux Networking ;-D

NIC Bonding
  • Active-active (balance-slb in XenServer provided by Open vSwitch)
  • Active-backup
  • balance-alb
  • balance-tlb
  • LACP Bond with load balancing based on IP and port of source and destination, or based on source MAC addresses
Jumbo Frame for IP-based Storage Traffic (iSCSI, NFS).

net-tools (the collection of base networking utilities for Linux)
iproute2 (a collection of utilities for controlling TCP/IP networking and traffic control in Linux)

Netfilter / iptables / conntrack{,d} - power user
NOTE: nftables (successor of iptables in kernel mainline since 3.13)

Linux Bridge (bridge-utils)
Open vSwitch - OVS (network backend of XenServer, virtual switch providing NIC Bonding, VLANs, QoS and etc..)
Floodlight - OpenFlow controller and Citrix XenServer DVSC (Nicira)

Commonly used CLI utilities
  • nmap
  • tcpdump (network traffic dumper in CLI)
  • Wireshark (AKA Ethereal) / tshark command line utility
  • ntop / netstat / ss / mtr / iftop / iptraf / route / ethtool / ethstatus / slurm / nethogs / bwm-ng / sar (sysstat) / lsof / dig / nslookup / host / ngrep ...
  • curl / wget / wput / aria2
  • iperf (iperf 2.x and iperf3, TCP/UDP/SCTP Bandwidth Measurement Tool - love it!)
  • netcat (nc)
  • netstat / ss
  • tc (traffic control)
  • OpenSSH (ssh, sshd, ssh-add, ssh-agent, ssh-copy-id, ssh-keygen, ssh-keyscan, scp, sftp
  • openssl
  • gpg / gpg2 (GnuPG - GNU's OpenPGP implementation)
  • mitmproxy / mitmdump
  • ...

Networking Protocols: SSH (OpenSSH implementation), HTTP (including HTTP/2), TCP/IP, SSL/TLS, IPsec, IKE, DHCP, SMTP/ESMTP, DNS, NFS, SMB, ICMP, BGP.

Switching and Routing basics, NAT, etc.

DevOps / Automation / Configuration Management

Chef
Chef Solo, Chef Client (knife, knife-solo), Chef Server.
Bootstrap (shell) scripts: chef-solo-bootstrap
chef-solo-skeleton project (to be added to GitHub)

Chef related tools
  • Berkshelf to manage cookbooks and their dependencies
  • Foodcritics - lint tool

Vagrant - Tool for building and distributing development environment (perfect with Chef Solo)
A list of base boxes for vagrant => vagrantbox.es Oracle Linux 7.2 x86_64, Oracle Linux 6.7 x86_64 & Oracle Linux 5.11 x86_64 base boxes were contributed by me;-)

Vagrant Plugins (hands-on experience)
  • vagrant-omnibus (install chef client for base boxes)
  • vagrant-berkshelf (manage cookbook dependencies for provision - bundler for cookbooks)
  • vagrant-salt
  • vagrant-digitalocean
  • vagrant-lxc
  • vagrant-httpproxy
  • vagrant-proxyconf (set proxy for APT)
  • vagrant-vbguest
NOTE: I answer Vagrant, VirtualBox, XenServer, Networking and Linux related questions on Stack Overflow ;-)

TerraformTerraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions.
  • Infrastructure as code
  • Execution Plans
  • Resource Graph
  • Change Automation

Packer - similar to Veewee but does not rely on Ruby, more light-weight and flexible (JSON + Shell scripts).

Docker / containerd
rkt

LXC / LXD

Kubernetes (Production grade container scheduling and management)
Minikube
Fission - FaaS (Function as a Service) for Kubernetes

Cloud Foundry

Veewee - Amazing tool to automate the building of custom Vagrant base boxes. Create definitions (shell scripts + definitions in Ruby) and build! The Oracle Linux {5,6,7} templates were contributed by me ;-)

Ad Hoc Tasks, Parallel execution, Administration and Deployment over SSH
  • Ansible (mainly parallel execution tool for me. Also playbooks, configuration management, deployment, and orchestration.)

Jenkins (A Hudson fork, Java CI - Continuous Integration tool, for build automation)

Knowledge Management (Wiki) & Issue Tracker

  •  Confluence (started from 2.0, all the way up to 5.1.3)
  • gollum (Git + Markdown + Sinatra)
  • Jekyll / Octopress (Vim + Git + Markdown)
  • JIRA as issue tracker (hands-on administration skills from 3.12.1 to 4.2)
  • Mediawiki (fuck it)

SCM (Source Code Management) / Version Control

  • Git - git power user
    Best version control - the most important invention by Linus Torvalds ) / GitHub (social coding;-)
  • GitLab => Self hosted Git Management Software (since version 4.0)
  • SVN - basic working knowledge
  • CVS (self hosted CVS Windows using CVSNT which is no longer free now. Forget about CVS, move on!)

Programming / Scripting

Shell Scripting (Bash)

Python (read, modify, debugging) - proficient working knowledge
NOTE: XenServer core XenAPI and utilities are heavy user of shell/python scripts. Some core components like sm (storage manager) are written in Python (83.1%) and Shell (13.1%).

Ruby
RubyGems, RVM, rbenv + ruby-build
Beginner, started to learn Ruby when coming across Homebrew (package manager for OS X), OpsCode Chef and Jekyll + Octopress , Sinatra (web application framework/lib, DSL, similar to the well known rails). 

Editors / IDE
  • Vim (Vundle.vim as plugin manager, moving to neovim soon)
  • Sublime Text {2,3}
  • RubyMine 4/5/6 (Ruby IDE)
diff / merge tool
  • vimdiff
  • meld (GUI)

JavaScriptJSON (used by Chef Cookbooks, Chef Server, MongoDB)

Markdown - heavy user for technical writing / documenting. Pandoc as universal document converter.

C
Capable of reading (crash dump analysis work - tracing down to kernel source code for RCA) , core dump related, strace (Linux system call related stuff), modifying and compiling C codes using GCC (LLVM + CLANG on OS X).

Java SE/Core Java (Swing/Socket//Networking/JDBC etc...)
Java SE and JVM Troubleshooting and Performance tuning (Sun JVM and JRockit Mission Control - now a part of Java SE 8).
Tools - Profiler/Performance Tuning Tools
JProfiler, YourKit Java Profiler, Eclipse Memory Analyzer (MAT, formerly SAP Memory Analyzer), IBM HeapAnalyzer, Samurai (Thread Dump viewer), ThreadLogic (Oracle), jconsole, jvisualvm.

Java IDEs
  • Eclipse (Window Builder, egit)
  • JBuilder
  • Oracle Enterprise Pack for Eclipse (OEPE) 11g

Cloud

  • AWS
    EC2, EBS, EFS, S3, VPC, VPN, ELB, RDS, IAM, Router 53 (DNS).
  • Citrix Accelerite CloudPlatform / Apache CloudStack
  • Cloud Foundry (with its micro VM by VMware)
  • Heroku (PaaS)
  • Google Apps (SaaS)
  • Iaas/VPS - RAM Host (OpenVZ) / BuyVM (OpenVZ) / 123Systems (Xen and KVM) / Digital Ocean (KVM)

Virtualization

  • VirtualBox {2,3,4,5}
  • VMware Workstation {3,4,5,6,7,8,9,10} (since version 3, 2001)
  • VMware OVF Tool (CLI)
  • VMware Server {1,2} (formerly GSX Server, now freeware)
  • VMware ESXi Server 4.0+
  • LXC/LXD
  • Docker
  • Xen Hypervisor 4.x
  • Citrix XenServer {6.0,6.0.2,6.1,6.2,6.5,7.0} => Black Belt
    vGPU with NVIDIA GRID K1/K2, PCI Pass-through, GPU Pass-through, USB Pass-through (with both xl pci-* XAPI and libvirt + virsh).
  • KVM/QEMU (with libvirt & virt-manager GUI)
  • CoreOS (Linux Kernel + systemd + LXC + Docker + Btrfs)
Parallels
Microsoft Virtual PC/Virtual Server/Hyper-V

Caching/Cache

  • Varnish Cache (HTTP Accelerator, Reverse Caching Proxy, Server Side cache/proxy)
  • Nginx as reverse proxy and cache server
  • Squid (Proxy Server, Forward Proxy) / squid-deb-proxy for Debian / Ubuntu (cannot get this shit to work, don't bother)
  • Memcached (in-memory key-value store)
  • Redis (in-memory key-value data store, support more types)

CDN - Content Delivery Network

CloudFlare / AWS CloudFront

Web Server/Application Server

  • Apache HTTP Server (httpd) {1.3.x,2.0.x,2.2.x,2.4.x} + mod_pagespeed
  • Nginx 0.7.x - 1.12.x (optional ngx_pagespeed) tracking stable and mainline
    Personal favourite web server, have been using it to serve personal web site and various applications since it's 0.7.0 release.
    Hands-on performance tuning, security hardening (securing traffic using SSL/TLS certificate, HSTS, etc.) experience.
    NOTE: My personal web sites are rated A+ by SSL Lab's SSL Server Test ;-D
  • Microsoft IIS {5.0,6.0,7.0}
  • Apache Tomcat {3,4,5,6,7,8}
  • Websphere Application Server 5/6
  • Oracle Web Tier (HTTP Server + Web Cache + OPMN - Oracle Process Manager and Notification Server)
  • Varnish (Web Cache, HTTP Accelerator, Reverse Proxy, Server side cache/proxy)
  • Jetty (light-weight HTTP web server and Java Servlet container)
  • Cherokee
  • HTTP File Server (HFS)

Load Balancing

  • HAProxy - Layer 7 (application layer) TCP and HTTP Load Balancer
    User space, slower but more flexible, easier to customize and tweak.
  • LVS (Linux Virtual Server) - Layer 4 (IP packets and UDP datagrams)
    Built-in in kernel, fast, but not very flexible.
  • nginx - layer 7 (mainly HTTP)
  • keepalived (VRRP implementation for virtual router failover and load balancing)


Infrastructure

  • OpenSSH
    Have been closely following OpenSSH since 6.7, 8 years+ experience dealing with OpenSSH.
    Safe and secure with Forward Secrecy (PFS) provided by Diffie-Hellman key exchange, which the NSA hates.
    Make sure you check this example ~/.ssh/config [Gist] NOTE: sshd configuration required for it to work.
    Powered by OpenSSH Manuals (RTFM), and Linux Networking Cookbook and SSH (The Secure Shell) - The Definitive Guide.
    Linux Ninja cannot live without this thing. Not only a secured communication channel,  but also port forwarding (including X11 Forwarding),  dynamic (application level) port forwarding (creates a SOCKS proxy), can be used to work around firewalls (including GFW, that when I started digging into SSH).
  • MTA {Postfix,Exim}, Mail Server Solution - Zimbra Collaboration Server,  
  • NFS {v3,v4}
  • SMB / CIFS using Samba (re-implementation on Linux)
  • VPN
    PPTP - PPP between hosts via GRE. MPPE encryption and MPPC compression, MS-CHAP v2 security. Solution for Linux: pptpd + iptables (do NAT).
    L2TP/IPsec - xl2tpd (establish the tunnel) + openswan/libreswan for encryption

    IPsec VPN - strongSwan, some of my early >>>strongSwan (IPsec VPN) instances<<< ;-D
    NOTEcapable of Install (compile from source) and configure strongSwan on Linux from scratch for remote access, Site-to-Site and Host-to-Host scenarios. Client configurations on most modern operating systems, deep understanding of IKEv1/IKEv2 and IPsec. Also familiar with ipsec-tools (Racoon IKE daemon).
    OpenVPN (SSL VPN, use OpenSSL for encryption, fast, highly configurable)
  • FTP proftpd/vsftpd (Linux) | Serv-U FTP Server/IIS (Windows) | Forget FTP, please use SFTP instead!!!
  • DNS (bind/named, dnsmasq, Amazon Route 53)
    Working knowledge of SPF, DNSSEC, DKIM and DMARC.
  • LDAP
  • DHCP
  • NTP (now bloody systemd takes care of NTP...)
  • TFTP (not useful any more as iPXE supports HTTP, NFS, FCoE, iSCSI etc, if TFTP is really needed for legacy PXE, dnsmasq does the job)
  • Firewall / Netfilter (iptables / nftables)
    iptables power user and practitioner (thanks to the GFW!?) ;-D

LDAP/Active Directory/Identity Management

  • Apache Directory Server / Studio
  • OpenLDAP 2.x
  • Oracle Internet Directory 11g
  • Sun Java System Directory Server 5.2 (AKA iPlanet DS, Netscape DS, now Oracle Directory Server EE)
  • Novell eDirectory 8.8 SP5
  • Microsoft Active Directory 2000/2003

Administration Tools: Apache Directory Studio, OpenLDAP client CLI -> ldap{search,add,delete,modify,passwd}

Single Sign-on/SSO

  • Windows Integrated Authentication (WNA, WIA)
  • Oracle Access Manager 11g (OHS + WebGate + WebLogic Server 11g + WebCenter 11g + OAM11g + OID 11g)

High Availability / Clustering / Disaster Recovery

WebLogic Server Cluster
WebLogic Portal Cluster Architecture (Zero downtime Architecture)
WebCenter 11g High Availability (clustering, failover)
Oracle Web Tier (OHS) Runtime Clustering
DRBD - (Network based RAID 1) - typically used for MySQL (DRBD + Pacemaker + Corosync) replicating over network, block device level replication.
keepalived (VRRP)
XenSever High Availability / DR
Apache Cassandra Cluster (Datastax OpsCenter as management tool)

RDBMS

  • Oracle Database 10gR1/10gR2
  • Oracle Database 11gR1/11gR2 (RAC)
  • MySQL {4.x,5.x} (I've been managing LAMP environment since Fedora Core 1 era)
    DBA, backup/restore, replication, High Availability using DRBD + Pacemaker + Corosync.
  • PostgreSQL
  • SQLite 3 GUI Tools: sqliteman
  • PointBase (embedded in WebLogic Server version, 10.3.2.0 and earlier. Since 10.3.3.0 it is replaced by Derby)

NoSQL (Not Only SQL) who cares about this shit any more?

  • Apache Cassandra (Install / Configure / Clustering / Monitoring)
    Hands on experience. Familiar with monitoring tools like DataStax OpsCenter Community Edition (free), jconsole and jvisualvm (cassandra-webconsole crap...).
  • MongoDB (Backup & Restore, Administration, Replication - Replica Set, Sharded Cluster)

Monitoring / Measuring / Management Tools

monit
supervisord / supervisor
monitorix
cacti
nagios
MRTG
Munin
StatsD (node.js powered) + Graphite
Ganglia
ntop
webmin
weblizer
gnuplot

Application Performance Management - APM
New Relic
Server Density

Log Management
Graylog2
logstash
goaccess

Other Free Open Source Software/Others

LAMP (Linux+ Apache + MySQL + PHP)
LEMP (Linux + Nginx + MySQL + PHP - php-fpm)
OpenSSL
GnuPG / OpenPGP
Keybase
Ksplice / kpatch
DataStax OpsCenter for Cassandra (Big Data)
mod_pagespeed for Apache (by Google)
XAMPP
Apache JMeter (load test)
Apache Tomcat
Discourse (Ruby on Rails discussion forum)
Discuz! (php)
WordPress (fuck it)
Octopress (Blogging framework for Hackers)
Jekyll
rbenv + ruby-build
homebrew
MovableType
phpMyAdmin / phpPgadmin
phpsysinfo
AWstats
Gregarius
......

Middleware (Portals/Content/SOA/BPM)


Oracle Fusion Middleware 11gR1 (11.1.1.x)

  • Repository Creation Utility
  • WebLogic Server
  • WebCenter Portal
  • WebCenter Content (formerly UCM/ECM, AKA Stellent) 10gR3/11g
  • SOA Suite (BPM, Service Bus)
  • Identity Management (OID/OVD, Access Manager - SSO)
  • Oracle Enterprise Pack for Eclipse (OEPE, AKA WebLogic Workshop or Workshop for WebLogic)

Oracle WebCenter Interaction {5.x,6.0.x,6.1.x,6.5.x,10.3.0.x,10.3.3.x} (AquaLogic User Interaction SuiteALUIPlumtree, WCI)

WebLogic Server 6, 7, 8.1.x, 9.x, 10.x, 10gR3 (10.3.0.0), 11gR1 (10.3.2.0 up to 10.3.6.0), 12c (12.1.1)

Oracle Certified Associate - WebLogic Server 11g System Administration (1Z0-102)

WebLogic Portal {10,10gR3,10.3.2,10.3.4,10.3.5}

Oracle Secure Enterprise Search {10gR1,11.1.2,11.1.2.2.0}

Oracle BPM (Fuego BPM, ALBPM) 5.5/5.7.x/6.0.x/10gR3 (I have no experience with 11g as it is JDeveloper based now-_-)
Oracle JRockit Mission Control 4.0 (JRockit JVM)


Comments