Databricks Community

The Help Center provides an additional forum for this topic. You can request a voucher by submitting a help request, however, vouchers are not provided in all cases. Other ways to obtain a voucher are participation in training events held throughout ...

Hi,Source table is external table in UC and  result table should be also external. I'm running such command CREATE TABLE target_catalog.target_schema.table_clone SHALLOW CLONE source_catalog.source_schema.source_table but this for some reason doesn't...

Also, does it mean that even I specify a column of type TimestampNTZ in the clustering key, it is not clustered by this column?

Read all the files using auto loader and add an additional column as follows:.withColumn("filePath",input_file_name())Now that you've file name, you can split the data frame as per your requirement and ingest data into different tables.

@Kaniz_Fatma I updated my code as below: df = spark.read.format("com.mongodb.spark.sql.DefaultSource")\ .option("database", database)\ .option("collection", collection)\ .option("spark.mongodb.input.uri", connectionString)\ ...

@arunak - we need to specify forward_spark_s3_credentials to true during read. This will help spark detect the credentials used to authenticate to the S3 bucket and use these credentials to r read from redshift.  

Hi @mh_db - you can import botocore library (or) if it is not found can do a pip install botocore to resolve this. Alternatively, you can maintain the data in a spark dataframe without converting to a pandas dataframe and while writing to a csv. you ...