如何写出优雅的 Golang 代码
原文是一篇非常赞的工程总结,推荐。这里记录一些笔记
- 代码规范,
gofmt
,goimports
,golangci-lint
等,配合 CI 自动化 - 目录结构 Standard Go Project Layout
/pkg
可以被外部使用包模块/internal
私有模块,不可被外部使用- 不要有
/src
目录 /cmd
生成可执行文件/api
对外提供的 API 模块- 不要有
model/controller
这样的模块,按照职责拆分,面向接口开发
- 不要在
init
做资源初始化,比如 rpc/DB/Redis 等,因为 init 会被隐式执行,会默默初始化资源连接 - 推荐的做法是 Client + NewClient,显式初始化连接
- init 阶段适合做一些简单、轻量的前置条件判断,比如
flag
设置 - 使用 GoMock/sqlmock/httpmock/monkey 做 mock + 测试
Simple techniques to optimise Go programs 介绍了一些简单却非常高效的性能提升方法:
- Avoid using structures containing pointers as map keys for large maps, use ints or bytes
- Use strings.Builder to build up strings
- Use strconv instead of
fmt.Sprintf
Power of g in Vim
:[range]g[!]/pattern/cmd
!
means do not match pattern, cmd list:
d
: deletem
: movet
: copy, orco
s
: replace
for more info:
:help ex-cmd
- Power of g
动态添加/删除 Hadoop DataNode
添加节点
- NameNode 添加节点
etc/hadoop/slaves
- 同步
etc/hadoop
配置 - 在新节点
./sbin/hadoop-daemon.sh start datanode
- 在 NameNode 刷新
hdfs dfsadmin -refreshNodes
删除节点
- 在
etc/hadoop/excludes
写入要删掉的节点地址 - 修改
etc/hadoop/hdfs-site.xml
:
<property>
<name>dfs.hosts.exclude</name>
<value>/home/web/hadoop/etc/hadoop/excludes</value>
</property>
- 修改
etc/hadoop/mapred-site.xml
, 这个是下线 nodemanager
<property>
<name>mapred.hosts.exclude</name>
<value>/home/web/hadoop/etc/hadoop/excludes</value>
<final>true</final>
</property>
- 修改
etc/hadoop/slaves
,去掉要删除的节点 - 同步
etc/hadoop/excludes
和etc/hadoop/slaves
到所有 NameNode - 在 NameNode 执行
hdfs dfsadmin -refreshNodes
hdfs dfsadmin -report
查看要删除的节点状态变化Normal -> Decommission in progress -> Decommissioned
- 在要删除的节点
./sbin/hadoop-daemon.sh stop datanode
,等待 Admin State 变更为 Dead
检查节点
hdfs fsck /
检查文件系统信息,正常是 Status: HEALTHY
,如果 Status: CORRUPT
说明 blocks 有损坏,其中 Missing blocks
表示有丢失,但有备份,Missing blocks (with replication factor 1)
表示 block 损坏丢失也没有备份,不可恢复。
可以用 hdfs fsck / -delete
来检查并删除有损坏的 blocks.
调整 JournalNode
- 修改
etc/hadoop/hdfs-site.xml
:
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop01:8485;hadoop02:8485;hadoop03:8485/cluster</value>
</property>
- 同步到所有节点
- 如果是新增节点,要同步
dfs.journalnode.edits.dir
下 edits 文件 - 在调整的 journalnode 节点启动/关停:
./sbin/hadoop-daemon.sh start journalnode
- 重启 standby NameNode:
sbin/hadoop-daemon.sh stop|start namenode
- 切换节点为 active:
hdfs haadmin -failover nn1 nn2
,重启其他 namenode - 检查 NN 状态
hdfs haadmin -getServiceState nn1
调整 NodeManager
- 修改
etc/hadoop/yarn-site.xml
<property>
<name>yarn.resourcemanager.nodes.exclude-path</name>
<value>/home/pubsrv/hadoop/etc/hadoop/excludes</value>
</property>
- 同步到 ResourceManager
- 重启 ResourceManager,
sbin/yarn-daemon.sh stop|start resourcemanager
- 修改 excludes,添加要删除的节点地址
yarn rmadmin -refreshNodes
yarn node -list -all
检查
调整 Spark 节点
- 新增:在 worker 上
./sbin/start-slave.sh spark://master:7077
- 删除: 在 worker 上
./sbin/stop-slave.sh
,需要注意的是如果$SPARK_PID_DIR
没有指定的话,默认是在/tmp
,类似/tmp/spark-hadoop-org.apache.spark.deploy.worker.Worker-1.pid
-
在 master 节点修改
conf/slaves
- https://www.jianshu.com/p/727da7ba438a
- https://www.iteye.com/blog/shift-alt-ctrl-2102571
Using special SSH key for Git
In ~/.ssh/config
:
host github.com
HostName github.com
IdentityFile ~/.ssh/id_rsa_github
User git
don’t forget chmod 600 ~/.ssh/config
Or, use GIT_SSH_COMMAND
environment variable:
export GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa_example -F /dev/null"
Regex Unicode Scripts
\p{Han}
匹配中文、日语文字,支持简繁体。\p{Common}
匹配符号\p{Latin}
匹配拉丁语系- 需要 grep perl 支持,即
grep -P "\p{Han}"
,或者rg/ag
.
echo '中文/繁體/片仮名/かたかな/カタカナ/katakana' | rg "\p{Han}" > 中文 繁體 片仮名
echo '中文@mail.com' | rg "\p{Common}" > @ .
echo '中文@mail.com' | rg "\p{Latin}" > mail com
Unicode Scripts for more.
Octotree for Safari
From Safari 13, you can only install extension from Mac App Store.
brew install node@10
export PATH="/usr/local/opt/node@10/bin:$PATH"
# make sure node and npm is v10, cause octotree used gulp 3, which is not working with node 12.
git clone https://github.com/ovity/octotree.git ~/src/octotree
cd ~/src/octotree
git checkout master
npm i
npm install [email protected]
npm run dist
# extension locate in ~/src/octotree/tmp/safari/octotree.safariextension/
cd ~/Library/Safari/Extensions
mv ~/src/octotree/tmp/safari/octotree.safariextension .
- Enable
Developer
menu in Safari Developer - Show Extension Builder
- Add octotree.safariextension and Run
MySQL Prefix Index
CREATE TABLE `t1` (
`bundle` varchar(300) DEFAULT '' COMMENT 'pkg name',
`domain` varchar(200) DEFAULT '',
UNIQUE KEY `idx_bundle_domain` (`bundle`(100),`domain`(100))
) ENGINE=InnoDB AUTO_INCREMENT=12 DEFAULT CHARSET=utf8mb4;
关键部分 bundle(100)
来解决组合索引可能会出现的 Specified key was too long; max key length is 767 bytes
错误。
Deployment with git
#!/bin/sh
set -uex
PATH=$PATH:$HOME/bin
export PATH
DIR=/home/serv/project
cd ${DIR}
REV1=$(git rev-parse --verify HEAD)
git pull origin master
REV2=$(git rev-parse --verify HEAD)
test ${REV1} = ${REV2} && echo "Already updated" && exit
make
test $? -ne 0 && echo "make error" && exit
kill -HUP $(cat logs/run.pid)
主要是通过 git rev-parse --verify HEAD
来获取当前 rev hash,前后对比是否一致,以此来决定是否继续。
logrotate
logrotate - rotates, compresses, and mails system logs
# 0 0 * * * /usr/sbin/logrotate --state=/home/serv/logrotate.state /home/serv/logrotate.log.conf
/home/serv/logs/dev.log
/home/serv/logs/access.log {
rotate 10
daily
compress
create
copytruncate
missingok
dateext
dateformat -%Y-%m-%d
dateyesterday
sharedscripts
postrotate
kill -USR1 `cat /var/run/nginx.pid`
endscript
}
- 要么保存到 /etc 配置,由系统调度。也可以自己通过 crontab 调度控制,这种情况要注意加
--state
来保存状态 - 像 nginx 可以通过
kill -USR1
来重新打开日志文件,如果服务不支持可以用copytruncate
,先拷贝再清空
Druid Query in JSON
Druid 可以在 Superset SQL 查询,除此之外可以通过 HTTP+JSON 查询:
curl -X POST '<host:<port>/druid/v2/?pretty' -H 'Content-Type:application/json' -H 'Accept:application/json' -d @query.json
{
"queryType": "timeseries",
"dataSource": "cpm_log",
"granularity": "hour",
"aggregations": [
{
"type": "longSum",
"name": "requests",
"fieldName": "req_count_raw"
},
{
"type": "longSum",
"name": "impressions",
"fieldName": "win_count"
},
{
"type": "floatSum",
"name": "revenues",
"fieldName": "win_price"
}
],
"postAggregations": [
{
"type":"arithmetic",
"name": "ecpm",
"fn": "/",
"fields": [
{
"type": "fieldAccess",
"name": "postAgg_rev",
"fieldName": "revenues"
},
{
"type": "fieldAccess",
"name": "postAgg_imps",
"fieldName": "impressions"
}
]
}
],
"filter": {
"type": "and",
"fields": [
{
"type": "selector",
"dimension": "device_os",
"value": "android"
},
{
"type": "in",
"dimension": "req_ad_type",
"values": ["banner"]
}
]
},
"context": {
"grandTotal": true
},
"intervals": [
"2019-04-09T00:00:00+08:00/2019-04-09T23:00:00+08:00"
]
}
- queryType 有
timeseries
,topN
,groupBy
,search
,timeBoundary
等 - 尽量少用 groupBy 查询,效率不高
- topN 查询是通过
metric
来排序 context
可以指定queryId
,这样可以通过DELETE /druid/v2/{queryId}
取消查询- 去重:
{"type": "cardinality", "name": "distinct_pid", "fields": ["ad_pid"]}