- 1117 Views
- 1 replies
- 0 kudos
Not able to connect with Salesforce, We need to read data from Salesforce
Not able to connect with Salesforce, We need to read data from Salesforce, we are getting NoClassDefFoundError: scala/Product$classCode:%scalaval sfDF = spark. read. format("com.springml.spark.salesforce"). option("username", "sf...
- 1117 Views
- 1 replies
- 0 kudos
- 0 kudos
@Amar.Kasar :The error you are getting, NoClassDefFoundError: scala/Product$class, suggests that the Scala classpath is not set up correctly. You can try the following steps to troubleshoot the issue:Check if the library com.springml:spark-salesforc...
- 0 kudos
- 9337 Views
- 1 replies
- 0 kudos
Pull query that inserts into table
I'm trying to pull some data down for table history and am needing to view the query that inserted into a table. My team owns the process so I'm able to view the current query by just viewing it but I'm also wanting to capture changes over time witho...
- 9337 Views
- 1 replies
- 0 kudos
- 0 kudos
@Coleman Milligan :Yes, in Databricks, you can use the built-in Delta Lake feature to track the history of changes made to a table, including the queries that inserted data into it.Here's an example of how to retrieve the queries that inserted data ...
- 0 kudos
- 8020 Views
- 5 replies
- 6 kudos
Resolved! How to make Autoloader delete files after a successful load
Hi AllCan you please advise how I can arrange loaded file deletion from Azure Storage upon its successful load via Autoloader? As I understood, Spark streaming "cleanSource" option is unavailable for Autoloader, so I'm trying to find the best way to ...
- 8020 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi @Artem Sachuk Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 6 kudos
- 8389 Views
- 2 replies
- 1 kudos
Resolved! Query data not showing in custom alert notifications and QUERY_RESULT_ROWS
I've set up a custom alert notification for one of my Databricks SQL queries, and it triggers correctly, but I'm not able to get the actual results of the query to appear in the notification email. I've followed the example/template in the custom ale...
- 8389 Views
- 2 replies
- 1 kudos
- 1 kudos
The actual query results are not displayed in the alert unfortunately. You can pass the alert condition etc, but not the raw results of the underlying query.I hope this will be added in the future.A workaround is to add a link to the query, so the r...
- 1 kudos
- 4975 Views
- 4 replies
- 3 kudos
Resolved! Streaming Delta Live Table, if I re-run the pipeline, does it append the new data to the current table?
Hi,I have a question about DLT table. Assume that I have a streaming DLT pipeline which reads data from a Bronze table and apply transformation on data. Pipeline mode is triggered. If I re-run the pipeline, does it append new data to the current tabl...
- 4975 Views
- 4 replies
- 3 kudos
- 3 kudos
@Mohammad Saber :In a Databricks Delta Lake (DLT) pipeline, when you re-run the pipeline in "append" mode, new data will be appended to the existing table. Delta Lake provides built-in support for handling duplicates through its "upsert" functionali...
- 3 kudos
- 2705 Views
- 3 replies
- 0 kudos
ODBC Connection to Another Compute Within the Same Workspace
Hello all!I couldn't find anything definitive related to this issue so I hope I'm not duplicating another topic :). I have imported an R repository that normally runs on another machine and uses ODBC driver to issue sparkSQL commands to a compute (le...
- 2705 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks @Suteja Kanuri for your response! I tried all of the steps you mentioned (and many more) but never managed to make it work. My suspicion was that our azure networking setup was preventing this from happening. I have not found this documented ...
- 0 kudos
- 3163 Views
- 1 replies
- 0 kudos
Where are delta lake files stored by given path?
I have below code which works for the path below but fails for path = azure storage account path. i have enough access to write and update the storage account. I would like to know what wrong am I doing and the path below which works , how can i phys...
- 3163 Views
- 1 replies
- 0 kudos
- 0 kudos
@Ankit Kumar :The error message you received indicates that the user does not have sufficient permission to access the Azure Blob Storage account. You mentioned that you have enough access to write and update the storage account, but it's possible t...
- 0 kudos
- 20165 Views
- 3 replies
- 1 kudos
How to extract values from JSON array field?
Hi,While creating an SQL notebook, I am struggling with extracting some values from a JSON array field. I need to create a view where a field would be an array with values extracted from a field like the one below, specifically I need the `value` fi...
- 20165 Views
- 3 replies
- 1 kudos
- 1 kudos
Maybe I didn't explain it correctly. The JSON snippet from the description is a cell from a row from a table.
- 1 kudos
- 3579 Views
- 2 replies
- 0 kudos
Getting Databricks SQL dashboard to recognise change to an underlying query
Hi CommunityScenario:I have created a query in Databricks SQL, built a number of visualisations from it and published them to a dashboard. I then realise that I need to add another field to the underlying query that I want to then leverage as a dashb...
- 3579 Views
- 2 replies
- 0 kudos
- 1267 Views
- 0 replies
- 0 kudos
Are training/ecommerce data tables available as CSVs?
The course "Apache Spark™ Programming with Databricks" requires data sources such as training/ecommerce/events/events.parquet. Are these available as CSV files? My company's databricks configuration does not allow me to mount to such repositories, bu...
- 1267 Views
- 0 replies
- 0 kudos
- 1641 Views
- 1 replies
- 0 kudos
Upgrading Ipython version without changing LTS version
I am using a specific Pydeeque function called ColumnProfilerRunner which is only supported with Spark 3.0.1, so I must use 7.3 LTS. Currently, I am trying to install "great_expectations" library on Python, which requires Ipython version==7.16.3, an...
- 1641 Views
- 1 replies
- 0 kudos
- 0 kudos
@Hitesh Goswami : please check if the below helps!To upgrade the Ipython version on a Databricks 7.3LTS cluster, you can follow these steps:Create a new library installation command using the Databricks CLI by running the following command in your l...
- 0 kudos
- 3948 Views
- 5 replies
- 0 kudos
Installing Bazel on databricks cluster
I am new to azure databricks and I want to install a library on a cluster and to do that I need to install bazel build tool first.I checked the site bazel but I am still not sure how to do it in databricks?I appriciate if any can help me and write me...
- 3948 Views
- 5 replies
- 0 kudos
- 0 kudos
Databricks migrated over from the standard Scala Build Tool (SBT) to using Bazel to build, test and deploy our Scala code. Follow this doc https://www.databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html
- 0 kudos
- 3349 Views
- 1 replies
- 1 kudos
Pandas DataFrame error when using to_csv
Hi Everyone, I would like to a Pandas Dataframe to /dbfs/FileStore/ using to_csv method.Usually it would just write the Dataframe to the path described but It has been giving me "FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStor...
- 3349 Views
- 1 replies
- 1 kudos
- 1 kudos
f = open("/dbfs/mnt/blob/myNames.txt", "r")
- 1 kudos
- 1624 Views
- 1 replies
- 0 kudos
Why are some of my AWS EBS volumes in my workspace unencrypted?
I noticed that 30GB of my EBS volumes are unencrypted, is there a reason for this, and is there a way to encrypt these volumes?
- 1624 Views
- 1 replies
- 0 kudos
- 0 kudos
https://docs.databricks.com/security/keys/customer-managed-keys-storage-aws.html#introductionThe Databricks cluster’s EBS volumes (optional) - For Databricks Runtime cluster nodes and other compute resources in the Classic data plane, you can option...
- 0 kudos
- 1558 Views
- 1 replies
- 2 kudos
Import paths using repos and installed libraries get confused
We use Azure Devops and Azure Databricks and have custom Python libraries. I placed my notebooks in the same repo and the structure is like this:mylib/ mylib/__init__.pyt mylib/code.py notebooks/ notebooks/job_notebook.py setup.pyAzure pipelines buil...
- 1558 Views
- 1 replies
- 2 kudos
- 2 kudos
It looks for the configs locally i suppose if you can share requirements .txt i can elaborate
- 2 kudos
| User | Count |
|---|---|
| 1614 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
