https://delta.io logo
#random
Title
n

Nagendra Darla

04/06/2023, 5:52 AM
Hi, I am following this document to verify multi cluster writes on Amazon EMR. https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3/ I do not see either the table or record getting created in DynamoDB. Am I missing anything? Any inputs are much appreciated. Below are my configurations
Copy code
SparkSession spark = SparkSession.builder()
        .config("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
        .config("fs.AbstractFileSystem.s3.impl", "org.apache.hadoop.fs.s3a.S3A")
        .config("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
        .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
        .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
        .config("spark.delta.logStore.s3.impl", "io.delta.storage.S3DynamoDBLogStore")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName", "delta_log")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.ddb.region", "us-east-1")
        .config("spark.io.delta.storage.S3DynamoDBLogStore.credentials.provider",
            "com.amazonaws.auth.DefaultAWSCredentialsProviderChain")
        .getOrCreate();
7 Views