hudi编译完成后与spark集成,集成后使用spark-shell操作hudi,怎么指定要使用hudi源的完全限定类名,这个错误怎么解决一下

maven对hudi进行构建,将spark-shell启动
spark-shell
--jars /root/hudi-jars/hudi-spark3-bundle_2.12-0.9.0.jar,
/root/hudi-jars/spark-avro_2.12-3.0.1.jar,/root/hudi-ja ars/spark_unused-1.0.0.jar
--conf "spark.serializer=org.apache.spark.serializer.Kr yoserializer"

img

不知道你这个问题是否已经解决, 如果还没有解决的话:
  • 这篇博客: 数据湖之Hudi(13):使用Spark删除Hudi中的数据中的 2. Maven依赖 部分也许能够解决你的问题, 你可以仔细阅读以下内容或者直接跳转源博客中阅读:

    在另一篇博文中有Maven依赖,但在这里还是补充一下

        <repositories>
            <repository>
                <id>aliyun</id>
                <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
            </repository>
            <repository>
                <id>cloudera</id>
                <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
            </repository>
            <repository>
                <id>jboss</id>
                <url>http://repository.jboss.com/nexus/content/groups/public</url>
            </repository>
        </repositories>
     
        <properties>
            <scala.version>2.12.10</scala.version>
            <scala.binary.version>2.12</scala.binary.version>
            <spark.version>3.0.0</spark.version>
            <hadoop.version>3.0.0</hadoop.version>
            <hudi.version>0.9.0</hudi.version>
        </properties>
     
        <dependencies>
     
            <!-- 依赖Scala语言 -->
            <dependency>
                <groupId>org.scala-lang</groupId>
                <artifactId>scala-library</artifactId>
                <version>${scala.version}</version>
            </dependency>
     
            <!-- Spark Core 依赖 -->
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_${scala.binary.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
            <!-- Spark SQL 依赖 -->
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_${scala.binary.version}</artifactId>
                <version>${spark.version}</version>
            </dependency>
     
            <!-- Hadoop Client 依赖 -->
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-client</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-common</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.hadoop</groupId>
                <artifactId>hadoop-hdfs</artifactId>
                <version>${hadoop.version}</version>
            </dependency>
     
            <!-- hudi-spark3 -->
            <dependency>
                <groupId>org.apache.hudi</groupId>
                <artifactId>hudi-spark3-bundle_2.12</artifactId>
                <version>${hudi.version}</version>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-avro_2.12</artifactId>
                <version>${spark.version}</version>
            </dependency>
     
        </dependencies>
     
        <build>
            <outputDirectory>target/classes</outputDirectory>
            <testOutputDirectory>target/test-classes</testOutputDirectory>
            <resources>
                <resource>
                    <directory>${project.basedir}/src/main/resources</directory>
                </resource>
            </resources>
            <!-- Maven 编译的插件 -->
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.0</version>
                    <configuration>
                        <source>1.8</source>
                        <target>1.8</target>
                        <encoding>UTF-8</encoding>
                    </configuration>
                </plugin>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.0</version>
                    <executions>
                        <execution>
                            <goals>
                                <goal>compile</goal>
                                <goal>testCompile</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
            </plugins>
        </build>

如果你已经解决了该问题, 非常希望你能够分享一下解决方案, 写成博客, 将相关链接放在评论区, 以帮助更多的人 ^-^