by
g96g
• New Contributor III
- 2956 Views
- 8 replies
- 0 kudos
I have project where I have to read the data from NETSUITE using API. Databricks Notebook runs perfectly when I manually insert the table names I want to read from the source. I have dataset (csv) file in adf with all the table names that I need to r...
- 2956 Views
- 8 replies
- 0 kudos
Latest Reply
Have you tried do debug the json payload of adf trigger ? maybe it wrongly conveys tables names
7 More Replies
- 9020 Views
- 6 replies
- 5 kudos
Hi
I am getting "java.lang.NoClassDefFoundError: scala/Product$class" error while using Deequ 1.0.5 version. Please suggest fix to this problem or any work around
Error
Py4JJavaError Traceback (most recent call last) <command-2625366351750561> in...
- 9020 Views
- 6 replies
- 5 kudos
- 1067 Views
- 4 replies
- 7 kudos
I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?
- 1067 Views
- 4 replies
- 7 kudos
- 2112 Views
- 3 replies
- 1 kudos
I have a Merge into statement that I use to update existing entries or create new entries in a dimension table based on a natural business key.When creating new entries I would like to also create a unique uuid for that entry that I can use to crossr...
- 2112 Views
- 3 replies
- 1 kudos
Latest Reply
you might wanna look into an identity column, which is possible now in delta lake.https://www.databricks.com/blog/2022/08/08/identity-columns-to-generate-surrogate-keys-are-now-available-in-a-lakehouse-near-you.html
2 More Replies
by
KVNARK
• Honored Contributor II
- 914 Views
- 3 replies
- 11 kudos
Is there any limitation in querying the no. of SQL queries in Databricks SQL workspace.
- 914 Views
- 3 replies
- 11 kudos
Latest Reply
1000 has been documented to be by default, though I have never checked the correctness.
2 More Replies
- 842 Views
- 2 replies
- 9 kudos
Hi allI want to integrate Kafka with databricks if anyone can share any doc or code it will help me a lot.Thanks in advance
- 842 Views
- 2 replies
- 9 kudos
Latest Reply
This is code that I am using to read from KafkainputDF = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", host)
.option("kafka.ssl.endpoint.identification.algorithm", "https")
.option("kafka.sasl.mechanism", "PLAIN")
.option("ka...
1 More Replies
- 4501 Views
- 8 replies
- 6 kudos
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 458.0 failed 4 times, most recent failure: Lost task 0.3 in stage 458.0 (TID 2247) (172.18.102.75 executor 1): com.databricks.sql.io.FileReadException: Error while rea...
- 4501 Views
- 8 replies
- 6 kudos
Latest Reply
Hi @Rupesh gupta​ Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.Cheers!
7 More Replies
- 2336 Views
- 2 replies
- 4 kudos
We are trying to connect to an Azure SQL Server from Azure Databricks using JDBC, but have faced issues because our firewall blocks everything. We decided to whitelist IPs from the SQL Server side and add a public subnet to make the connection work. ...
- 2336 Views
- 2 replies
- 4 kudos
Latest Reply
Using subnets for Databricks connectivity is the correct thing to do. This way you ensure the resources (clusters) can connect to the SQL Database. We also recommend using NPIP (No Public IPs) so that there won't be any public ip associated with the...
1 More Replies
- 4932 Views
- 4 replies
- 2 kudos
Hello,We have a business request to compare the evolution in a certain delta table.We would like to compare the latest version of the table with the previous one using Delta time travel.The main issue we are facing is to retrieve programmatically us...
- 4932 Views
- 4 replies
- 2 kudos
Latest Reply
In the docs it says that "'Neither timestamp_expression nor version can be subqueries." So it does sound challenging. I also tried playing with widgets to see if it could be populated using SQL but didn't succeed. With python it's really easy to do.
3 More Replies
- 989 Views
- 3 replies
- 4 kudos
I have a notebook that sets up parameters for the run based on some job parameters set by the user as well as the current date of the run. I want to supersede some of this logic and just use the manual values if kicked off manually. Is there a way to...
- 989 Views
- 3 replies
- 4 kudos
Latest Reply
You can create widgets by using this- dbutils.widgets.text("widgetName", "")To get the value for that widget:- dbutils.widgets.get("widgetName")So by using this you can manually create widgets (variable) and can run the process by giving desired valu...
2 More Replies
- 686 Views
- 1 replies
- 0 kudos
In a scala note, how to I read input arguments (e.g. those proved by a job that runs a scala notebook). In python, dbutils.notebook.entry_point.getCurrentBindings() works. How about for scala.
- 686 Views
- 1 replies
- 0 kudos
Latest Reply
Hi @Robert Russell​ You can use dbutils.notebook.getContext.currentRunId in scala notebooks. Other methods are also available likedbutils.notebook.getContext.jobGroupdbutils.notebook.getContext.rootRunId dbutils.notebook.getContext.tags etc...You ...
by
Snuki
• New Contributor II
- 318 Views
- 0 replies
- 0 kudos
Hi FOLKS, could you please guide me, why my points not reflecting in reward store, it is showing 0.
- 318 Views
- 0 replies
- 0 kudos
- 1958 Views
- 2 replies
- 0 kudos
I want to read password protected excel file and load the data delta table.Can you pleas let me know how this can be achieved in databricks?
- 1958 Views
- 2 replies
- 0 kudos
Latest Reply
df = spark.read.format("com.crealytics.spark.excel")\ .option("dataAddress", "'Base'!A1")\ .option("header", "true")\ .option("workbookPassword", "test")\ .load("test.xlsx")display(df)
1 More Replies
