- 2961 Views
- 4 replies
- 7 kudos
How to handle schema validation for Json file. Using Databricks Autoloader?
Following are the details of the requirement:1. I am using databricks notebook to read data from Kafka topic and writing into ADLS Gen2 container i.e., my landing layer.2. I am using Spark code to read data from Kafka and write into landing...
- 2961 Views
- 4 replies
- 7 kudos
- 7 kudos
Hi @Swapnil Kamle​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 7 kudos
- 4808 Views
- 9 replies
- 2 kudos
Resolved! need to remove doubledagger delimiter from a csv using databricks
My csv data looks like this‡‡companyId‡‡,‡‡empId‡‡,‡‡regionId‡‡,‡‡companyVersion‡‡,‡‡Question‡‡I tried this codedff = spark.read.option("header", "true").option("inferSchema", "true").option("delimiter", "‡,").csv(f"/mnt/data/path/datafile.csv")But I...
- 4808 Views
- 9 replies
- 2 kudos
- 2 kudos
Hi @shamly pt​ I took a bit another approach since I guess no one would be sure of the the encoding of the data you showed. Sample data I took :‡‡companyId‡‡,‡‡empId‡‡,‡‡regionId‡‡,‡‡companyVersion‡‡,‡‡Question‡‡‡‡1‡‡,‡‡121212‡‡,‡‡R‡‡,‡‡1.0A‡‡,‡‡NA‡‡...
- 2 kudos
- 2106 Views
- 2 replies
- 0 kudos
Terraform can set ALL_PRIVILEGES and USE_CATALOG on catalogs for 'account users', but not # SELECT or USE_SCHEMA
Only the GUI seems to allow SELECT and USE_SCHEMA 'account users' permissions on catalogs. Terraform gives me an error. Here is my Terraform config:resource "databricks_grants" "staging" { provider = databricks.workspace catalog = databricks_catalog....
- 2106 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Andrei Radulescu-Banu​ ,Which version of the provider are you using?I did check the github repo it should work:https://github.com/databricks/terraform-provider-databricks/blob/d65ef3518074a48e079080d94e1ab33a80bf7e0f/catalog/resource_grants.go#L1...
- 0 kudos
- 10583 Views
- 6 replies
- 8 kudos
Resolved! Is there some form of enablement required to use Delta Live Tables (DLT)?
I'm trying to use delta live tables, but if I import even the example notebooks I get a warning saying `ModuleNotFoundError: No module named 'dlt'`. If I try and install via pip it attempts to install a deep learning framework of some sort.I checked ...
- 10583 Views
- 6 replies
- 8 kudos
- 8 kudos
Here's the solution I came up with... Replace `import dlt` at the top of your first cell with the following: try: import dlt # When run in a pipeline, this package will exist (no way to import it here) except ImportError: class dlt...
- 8 kudos
- 2604 Views
- 4 replies
- 7 kudos
Resolved! Ingest Cosmos Mongo DB data using Databricks by applying filters
I would need to add a filter condition while ingesting data from a Cosmos Mongo DB using Databricks,I am using the below query to ingest data of a Cosmos Collection:df = spark.read \.format('com.mongodb.spark.sql.DefaultSource') \.option('uri', sourc...
- 2604 Views
- 4 replies
- 7 kudos
- 7 kudos
Hi @Swapnil Sarkar​, The error message means the stage name in your aggregation pipeline request wasn't recognised. The solution will be to ensure that all aggregation pipeline names are valid in your request.This article describes common errors and ...
- 7 kudos
- 3177 Views
- 3 replies
- 6 kudos
Resolved! pyspark - execute dynamically framed action statement stored in string variable
I need to execute union statement which is framed dynamically and stored in string variable. I framed the union statement, but struck with executing the statement. Does anyone know how to execute union statement stored in string variable? I'm using p...
- 3177 Views
- 3 replies
- 6 kudos
- 6 kudos
@Dineshkumar Gopalakrishnan​ using python's exec() function can be used to execute a python statement, which in your case could be pyspark union statement. Refer below sample code snippet for your reference.df1 = spark.sparkContext.parallelize([(1, 2...
- 6 kudos
- 2204 Views
- 1 replies
- 4 kudos
Importing Azure SQL data into Databricks
Hi,I am looking at building a data warehouse using Databricks. Most of the data will be coming from Azure SQL, and we now have Azure SQL CDC enabled to capture changes. Also I would like to import this without paying for additional connectors like Fi...
- 2204 Views
- 1 replies
- 4 kudos
- 4 kudos
@Bear Woods​ Hi! were you able to create DLT tables using CDC feature from sources like sql tables ? even I'm kinda in your situation, you need to leverage apply_changes function and create_streaming_live_table() function but it required intermediate...
- 4 kudos
- 5121 Views
- 8 replies
- 0 kudos
Resolved! ADF pipeline fails when passing the parameter to databricks
I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to r...
- 5121 Views
- 8 replies
- 0 kudos
- 0 kudos
Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names
- 0 kudos
- 10979 Views
- 6 replies
- 5 kudos
java.lang.NoClassDefFoundError: scala/Product$class
Hi I am getting "java.lang.NoClassDefFoundError: scala/Product$class" error while using Deequ 1.0.5 version. Please suggest fix to this problem or any work around Error Py4JJavaError Traceback (most recent call last) <command-2625366351750561> in...
- 10979 Views
- 6 replies
- 5 kudos
- 1748 Views
- 4 replies
- 7 kudos
Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.
I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?
- 1748 Views
- 4 replies
- 7 kudos
- 3308 Views
- 3 replies
- 1 kudos
Resolved! How to use uuid in SQL merge into statement
I have a Merge into statement that I use to update existing entries or create new entries in a dimension table based on a natural business key.When creating new entries I would like to also create a unique uuid for that entry that I can use to crossr...
- 3308 Views
- 3 replies
- 1 kudos
- 1 kudos
you might wanna look into an identity column, which is possible now in delta lake.https://www.databricks.com/blog/2022/08/08/identity-columns-to-generate-surrogate-keys-are-now-available-in-a-lakehouse-near-you.html
- 1 kudos
- 1483 Views
- 3 replies
- 11 kudos
Is there any limitation in querying the no. of SQL queries in Databricks SQL workspace.
Is there any limitation in querying the no. of SQL queries in Databricks SQL workspace.
- 1483 Views
- 3 replies
- 11 kudos
- 11 kudos
1000 has been documented to be by default, though I have never checked the correctness.
- 11 kudos
- 9808 Views
- 27 replies
- 58 kudos
- 9808 Views
- 27 replies
- 58 kudos
- 1560 Views
- 2 replies
- 9 kudos
Kafka integration with Databricks
Hi allI want to integrate Kafka with databricks if anyone can share any doc or code it will help me a lot.Thanks in advance
- 1560 Views
- 2 replies
- 9 kudos
- 9 kudos
This is code that I am using to read from KafkainputDF = (spark .readStream .format("kafka") .option("kafka.bootstrap.servers", host) .option("kafka.ssl.endpoint.identification.algorithm", "https") .option("kafka.sasl.mechanism", "PLAIN") .option("ka...
- 9 kudos
- 7282 Views
- 8 replies
- 6 kudos
Databricks notebook failed with "Caused by: java.io.FileNotFoundException: Operation failed: "The specified path does not exist.", 404, HEAD, https://adls.dfs.core.windows.net/raw/file.csv?upn=false&action=getStatus&timeout=90".
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 458.0 failed 4 times, most recent failure: Lost task 0.3 in stage 458.0 (TID 2247) (172.18.102.75 executor 1): com.databricks.sql.io.FileReadException: Error while rea...
- 7282 Views
- 8 replies
- 6 kudos
- 6 kudos
Hi @Rupesh gupta​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
- 6 kudos