- 5002 Views
- 5 replies
- 5 kudos
SQL IDE Support
How to use a SQL IDE with Databricks SQLDatabricks provides SQL IDE support using DataGrip and DBeaver with Databricks SQL.Let us know in the comments if you've used DataDrip or DBeaver with Databricks! Let us know if there are any other SQL IDEs you...
- 5002 Views
- 5 replies
- 5 kudos
- 5 kudos
dbeaver is perfectly working fine but I fount one issue it wont show the correct error for query.
- 5 kudos
- 4204 Views
- 4 replies
- 1 kudos
Resolved! Yml file replacing job cluster with all-purpose cluster when running a workflow
Hi all,I have been having some trouble running a workflow that consists of 3 tasks that run sequentially. Task1 runs on an all-purpose cluster and kicks off Task2 that needs to run on a job cluster. Task2 kicks off Task3 which also uses a job cluster...
- 4204 Views
- 4 replies
- 1 kudos
- 1 kudos
I don't know if you've cut off your yaml snippet, but your snippet doesn't show your job cluster with key job-cluster. Just to validate, your job cluster is also defined in your workflow yaml?Edit: Looking it it again and knowing the defaults, it loo...
- 1 kudos
- 11006 Views
- 7 replies
- 0 kudos
Resolved! How to import a function to another notebook?
Could you please provide guidance on the correct way to dynamically import a Python module from a user-specific path in Databricks Repos? Any advice on resolving the ModuleNotFoundError would be greatly appreciated.udf_check_table_exists notebook:fro...
- 11006 Views
- 7 replies
- 0 kudos
- 2218 Views
- 1 replies
- 0 kudos
Terraform Error: Cannot create sql table context deadline
I am currently trying to deploy external parquet tables to the Databricks UC using terraform. However, for some tables I get the following error:Error: cannot create sql table: Post "https://[MASKED]/api/2.0/sql/statements/": context deadline exceede...
- 2218 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @Retired_mod,thanks for your reply and sorry for the late reply from my side. I couldn't fix the problem with the databricks terraform provider unfortunately. I now switched to using liquibase to deploy tables to databricks.
- 0 kudos
- 3088 Views
- 4 replies
- 0 kudos
CREATE WIDGETS in SQL Notebook attached to SQL Warehouse Doesn't Work.
I'm able to create and use widgets using the UI in my SQL notebooks, but they get lost quite frequently when the notebook is reset.There is documentation suggesting we can create widgets in code in SQL: https://learn.microsoft.com/en-us/azure/databri...
- 3088 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @JeremyH - can you please try adding like the below in your query and see if widgets are getting populated? {{parameter_name }}
- 0 kudos
- 910 Views
- 0 replies
- 0 kudos
DatabricksUniForm
Hi Community members, I tried creating a Delta UniForm table using databricks notebook. I created a database without providing location. It took the dbfs default storage location. On top of that I was able to create a Delta UniForm table. Then I trie...
- 910 Views
- 0 replies
- 0 kudos
- 1627 Views
- 2 replies
- 0 kudos
Resolved! Unzipping with Serverless Compute
HiI have started using serverless compute but have come across the limitation that I cannot use the local filesystem for temporarily storing the files and directories before moving them to where they need to be in adls.Does anyone have a way of unzip...
- 1627 Views
- 2 replies
- 0 kudos
- 0 kudos
Do you have the ability to make a Unity Catalog volume? You could use it as temporary storage before migrating the files to adls.
- 0 kudos
- 1563 Views
- 0 replies
- 0 kudos
notebook returns succeeded even if there is a failure in any of the command
I'm trying to run a notebook that has 14 commands from adf notebook activity, I'm writing the transformed data to a delta table in the last command. I'm handling the last command in try except block to handle errors, and raising error on exception.wh...
- 1563 Views
- 0 replies
- 0 kudos
- 3279 Views
- 2 replies
- 0 kudos
Resolved! Databricks to Oracle to Delete Rows
Hi Community! I’m working in a Fatabricks notebook and using the Oracle JDBC Thin Client connector to query and Oracle table, merge together and select specific rows from my dataframe and write those rows to a table back in Oracle. All of this works...
- 3279 Views
- 2 replies
- 0 kudos
- 0 kudos
This stack overflow post ran into the same issue, would be worth trying How to delete a record from Oracle Table in Python SQLAlchemy - Stack Overflow
- 0 kudos
- 11290 Views
- 3 replies
- 2 kudos
Incremental load from source; how to handle deletes
Small introduction; I'm a BI/Data developer, mostly working in MSSQL & DataFactory, coming from SSIS. Now I'm trying Databricks, to see if it works for me and my customers. I got enthusiastic by the video; https://www.youtube.com/watch?v=PIFL7W3DmaY&...
- 11290 Views
- 3 replies
- 2 kudos
- 1801 Views
- 1 replies
- 0 kudos
Spark Scala Vs Pyspark
With the release of spark connect and used defined table functions for pyspark, I wonder, what are the remaining advantages (if any) of using scala Spark?
- 1801 Views
- 1 replies
- 0 kudos
- 0 kudos
The main remaining advantages of Scala are performance as there will always be some interoperation overhead when using PySpark. While I don't have any stats on me, I would assume the differences in performance are negligible at this point until very ...
- 0 kudos
- 3951 Views
- 2 replies
- 0 kudos
Databricks Managing Materialized Views in Delta Live Tables: Selective Refresh Behavior
Hi Community,I have 200 complex SQL Queries and I can't create a Streaming tables using these queries. So, I have created as Materialized Views in Delta Live Tables using these SQL queries and the DLT pipeline should be run continuously.My question i...
- 3951 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello team, thanks for reaching out us, it will be a pleasure to help you on this ask. That's a great catch to run through a materialized, view and regarding the question: If any record of underlying table is inserted, updated or deleted, the only re...
- 0 kudos
- 6151 Views
- 3 replies
- 3 kudos
Resolved! No userid, username, job when making modifications on tables
Hi, everyone!I'm in this situationI have some jobs that makes changes on a particular table. I use only one user to make this modifications, but then there's a process i can't identify that also makes changes on my table.The question is, there's a re...
- 6151 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi, @shan_chandra, @LuisRSanchez,Just found that there are some .jar that are executed and are writting on this table but this .jar is called through batchSo, we think this is the cause Thanks!Ingrid
- 3 kudos
- 1139 Views
- 1 replies
- 0 kudos
Databricks job cluster logs
Hello, how can I enable Databricks to generate a separate spark log for each job run?What parameters should I use in spark configuration?
- 1139 Views
- 1 replies
- 0 kudos
- 0 kudos
@Jackson1111 - If you are talking about workflow jobs, you can try running using a job cluster to generate spark logs for a each of the workflow jobs. But, If this is of Spark Jobs within the Spark UI, you wanted to separate out the logs. This is a ...
- 0 kudos
- 3460 Views
- 6 replies
- 1 kudos
Resolved! Azure Devops CI/CD - AWS Databricks
Hello, there is documentation for integrating Azure Devops CI/CD pipeline with AWS Databricks.Thanks.
- 3460 Views
- 6 replies
- 1 kudos
- 1 kudos
You'll need to install the AWS Toolkit in Azure DevOps, that way you can make a service connection inside your Azure DevOps project that authenticates using an AWS Access Key ID/Secret Access Key (the AWS equivalent to Azure Service Principals). Hope...
- 1 kudos
| User | Count |
|---|---|
| 1614 | |
| 773 | |
| 349 | |
| 286 | |
| 253 |
