SeaTunnel

安装JDK

参考</docs/linux/Centos&Oracle/安装JDK.md>

安装Maven

参考</docs/linux/Centos&Oracle/安装Maven.md>

下载安装文件及创建目录

创建seatunnel后端服务安装目录

mkdir -p /opt/bigdata/seatunnel-2.3.8/backend

创建seatunnel前端服务安装目录

mkdir -p /opt/bigdata/seatunnel-2.3.8/web

下载或者本地上传安装包

下载apache-seatunnel-2.3.8-bin.tar.gz

#进入1.3.2中创建好的安装目录
cd /opt/bigdata/seatunnel-2.3.8/backend
#下载安装包
wget https://dlcdn.apache.org/seatunnel/2.3.8/apache-seatunnel-2.3.8-bin.tar.gz

下载apache-seatunnel-web-1.0.0.tar.gz

#进入1.3.2中创建好的安装目录
cd /opt/bigdata/seatunnel-2.3.8/web

#下载安装包
wget https://dlcdn.apache.org/seatunnel/seatunnel-web/1.0.2/apache-seatunnel-web-1.0.2-bin.tar.gz

安装Apache Seatunnel

cd /opt/bigdata/seatunnel-2.3.8/backend
#解压后端安装包
tar -zxvf /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8-bin.tar.gz

#重命名安装包
mv apache-seatunnel-2.3.8-bin apache-seatunnel-2.3.8

cd /opt/bigdata/seatunnel-2.3.8/web

#解压前端安装包
tar -zxvf /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2-bin.tar.gz

#重命名安装包
mv apache-seatunnel-web-1.0.2-bin apache-seatunnel-web-1.0.2

配置环境变量

vim /etc/profile

在最后一行追加

export SEATUNNEL_HOME=/opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8
export PATH=${PATH}:${SEATUNNEL_HOME}/bin

刷新配置

source /etc/profile

下载connector jar包

创建目录

cd /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/connectors

自动下载

cd /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8
sh bin/install-plugin.sh

手动下载

先备份

cp -r /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/bin/install-plugin.sh /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/bin/install-plugin.sh.bak

把${SEATUNNEL_HOME}/mvnw改为mvn

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

#This script is used to download the connector plug-ins required during the running process. 
#All are downloaded by default. You can also choose what you need. 
#You only need to configure the plug-in name in config/plugin_config.

# get seatunnel home
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)

# connector default version is 2.3.8, you can also choose a custom version. eg: 2.3.8:  sh install-plugin.sh 2.3.8
version=2.3.8

if [ -n "$1" ]; then
    version="$1"
fi

echo "Install SeaTunnel connectors plugins, usage version is ${version}"

# create the connectors directory
if [ ! -d ${SEATUNNEL_HOME}/connectors ];
  then
      mkdir ${SEATUNNEL_HOME}/connectors
      echo "create connectors directory"
fi

while read line; do
    first_char=$(echo "$line" | cut -c 1)

    if [ "$first_char" != "-" ] && [ "$first_char" != "#" ] && [ ! -z $first_char ]
      	then
      		echo "install connector : " $line
      		#${SEATUNNEL_HOME}/mvnw dependency:get -Dtransitive=false -DgroupId=org.apache.seatunnel -DartifactId=${line} -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/connectors
            mvn dependency:get -Dtransitive=false -DgroupId=org.apache.seatunnel -DartifactId=${line} -Dversion=${version} -Ddest=${SEATUNNEL_HOME}/connectors
    fi

done < ${SEATUNNEL_HOME}/config/plugin_config

配置 SeaTunnel 同步作业

编辑 config/v2.batch.config.template 文件，该文件决定了在启动 SeaTunnel 后数据输入、处理和输出的方式及逻辑。

配置文件示例:

# 环境配置
env {
  # 指定任务是批模式还是流模式，job.mode = "BATCH"为批模式，job.mode = "STREAMING"为流模式
  parallelism = 2
  job.mode = "BATCH"
  checkpoint.interval = 10000
}

# 数据源
source {
  # 定义SeaTunnel从哪里获取数据。支持同时配置多个源，每个源都有特有的参数用于定义如何获取数据。
  FakeSource {
    parallelism = 2
    result_table_name = "fake"
    row.num = 16
    schema = {
      fields {
        name = "string"
        age = "int",
        card = "int"
      }
    }
  }
}

# 转换器
transform {
  Filter {
    source_table_name = "fake"
    result_table_name = "fake1"
    fields = [name, card]
  }
}

# 写入
sink {
  Console {
    source_table_name = "fake1"
  }
}

运行seatunnel服务

测试验证seatunnel

进入seatunnel后台服务目录

cd /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8

启动服务

./bin/seatunnel.sh --config ./config/v2.batch.config.template -e local

启动seatunnel服务

进入seatunnel后台服务目录

cd /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8

启动服务

nohup sh bin/seatunnel-cluster.sh 2>&1 &

查询启动日志

tail -f /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/logs/seatunnel-engine-server.log

安装seatunnel-web

进入seatunnel-web服务目录

cd /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/

初始化数据库

注意：下面这步要提前装好mysql（mysql>= 5.7.28），如果没有装干脆直接取script/seatunnel_server_mysql.sql到自己的mysql服务器上导入，后面配置从localhost改到自己的服务器

编辑seatunnel_server_env.sh文件，修改数据库地址、端口、账户、密码，并修改配置加上前缀STWEB_

vim script/seatunnel_server_env.sh

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

export STWEB_HOSTNAME="127.0.0.1"
export STWEB_PORT="3306"
export STWEB_USERNAME="root"
export STWEB_PASSWORD="123456"

vim script/init_sql.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

workDir=`dirname $0`
workDir=`cd ${workDir};pwd`

source ${workDir}/seatunnel_server_env.sh

usage="Usage: seatunnel_server_env.sh must contain hostname/port/username/password."

if [[ ! -n "${STWEB_HOSTNAME}" ]]  || [[ ! -n "${STWEB_PORT}" ]] || [[ ! -n "${STWEB_USERNAME}" ]] || [[ ! -n "${STWEB_PASSWORD}" ]]; then
    echo $usage
    exit 1
fi

mysql -h${STWEB_HOSTNAME} -P${STWEB_PORT} -u${STWEB_USERNAME} -p${STWEB_PASSWORD} < ${workDir}/seatunnel_server_mysql.sql

执行初始化数据库命令

sh script/init_sql.sh

修改端口与数据源

修改端口号和数据源信息

vim conf/application.yml

注：必须填写secretKey

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

server:
  port: 8801

spring:
  application:
    name: seatunnel
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
  datasource:
    driver-class-name: com.mysql.cj.jdbc.Driver
    url: jdbc:mysql://localhost:3306/seatunnel?useSSL=false&useUnicode=true&characterEncoding=utf-8&allowMultiQueries=true&allowPublicKeyRetrieval=true
    username: root
    password: 123456
  mvc:
    pathmatch:
      matching-strategy: ant_path_matcher

jwt:
  expireTime: 86400
  # please add key when deploy
  secretKey: ZkEa5pnp5eXq7/eSdTv6W4rw+lv4ywJ4xsAj7iK29MY=
  algorithm: HS256

---
spring:
  config:
    activate:
      on-profile: h2
  sql:
    init:
      schema-locations: classpath*:script/seatunnel_server_h2.sql
  datasource:
    driver-class-name: org.h2.Driver
    url: jdbc:h2:mem:seatunnel;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true
    username: sa
    password: sa
  h2:
    console:
      enabled: true
      path: /h2
      settings:
        trace: false
        web-allow-others: false

配置client信息

将seatunnel引擎服务节点的安装目录下的config目录下的关于引擎客户端的配置文件拷贝到seatunnel-web安装目录下的conf目录下同一台机器下部署直接使用以下拷贝命令(注意修改服务的安装目录为你自己的安装目录)

sudo cp /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/config/hazelcast-client.yaml /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/conf

如果不在同一台机器上，可以使用scp命令或者下载下来然后上传到web服务的安装主机的安装目录下的conf目录下即可。

配置支持的插件信息

将seatunnel引擎服务节点的安装目录下的connectors目录下的plugin-mapping.properties配置文件拷贝到seatunnel-web安装目录下的conf目录下

sudo cp /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/connectors/plugin-mapping.properties /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/conf

同一台机器下部署直接使用以下拷贝命令(注意修改服务的安装目录为你自己的安装目录)如果不在同一台机器上，可以使用scp命令或者下载下来然后上传到web服务的安装主机的安装目录下的conf目录下即可。

下载jar包

在线下载jar包，存在datasource文件夹

sh /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/bin/download_datasource.sh

下载mysql-connector-j-8.3.0.jar包，并放到/opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/libs 下地址：https://downloads.mysql.com/archives/c-j/

cp -r mysql-connector-j-8.3.0.jar /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/libs

启动seatunnel-web服务

cd /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2
sh bin/seatunnel-backend-daemon.sh start

# 停止
sh bin/seatunnel-backend-daemon.sh stop

# 查询状态
sh bin/seatunnel-backend-daemon.sh status

访问地址

Url:http://localhost:8081 账户：admin 密码：admin

未解决问题

额，访问是可以访问了，但是有个bug，选择整库同步时选不出来source，真的是奇葩

全拷贝到lib也无效

cp -r /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/datasource/* /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/libs
cp -r /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/datasource/* /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/lib

cp -r /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/connectors/* /opt/bigdata/seatunnel-2.3.8/web/apache-seatunnel-web-1.0.2/libs
cp -r /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/connectors/* /opt/bigdata/seatunnel-2.3.8/backend/apache-seatunnel-2.3.8/lib

安装JDK​

安装Maven​

下载安装文件及创建目录​

创建seatunnel后端服务安装目录​

创建seatunnel前端服务安装目录​

下载或者本地上传安装包​

安装Apache Seatunnel​

下载connector jar包​

创建目录​

自动下载​

手动下载​

配置 SeaTunnel 同步作业​

运行seatunnel服务​

测试验证seatunnel​

启动seatunnel服务​

安装seatunnel-web​

初始化数据库​

修改端口与数据源​

下载jar包​

启动seatunnel-web服务​

访问地址​

未解决问题​