- 2113 Views
- 3 replies
- 1 kudos
SQL Identifier clause
Hi, I was trying to prepare some dynamic SQLs to create table using the IDENTIFIER clause and WITH AS clause, but I'm stuck on some bug as it seems. could someone verify it or tell me that I am doing something wrong?code is running on SQL Warehouse T...
- 2113 Views
- 3 replies
- 1 kudos
- 1 kudos
Hey, I am struck with the same issue, did you find any workaround. I am trying to run DESCRIBE & ALTER command using IDENTIFIER() using databricks-sql-connector. Did u figure out how to run the identifier command statements??
- 1 kudos
- 583 Views
- 3 replies
- 0 kudos
Using WorkspaceClient -- run a saved query
I've saved a query on my sql warehouse which has a parameter called :list_parameter. I've found my query id as follows: from databricks.sdk import WorkspaceClient w = WorkspaceClient() for query in w.queries.list(): print(f"query: {query.displ...
- 583 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @lprevost The WorkspaceClient provides APIs to manage Query objects. But it doesn't provide the API to run it. If you need to run the query from a notebook, you can pass the query text into `spark.sql`. It returns SparkDataFrame. I hope this help...
- 0 kudos
- 695 Views
- 3 replies
- 1 kudos
Resolved! Streaming data - Merge in Target - DLT
We have streaming inputs coming from streaming tables and also the table from apply_changes.In our target there is only one table which needs to be merged with all the sources. Each source provides different columns in our target table. Challenge: Ev...
- 695 Views
- 3 replies
- 1 kudos
- 648 Views
- 2 replies
- 1 kudos
Resolved! DLT pipline with generated identity column
I got a csv file which I am looking to read into a streaming table. I always want to add a generated identity column as surrogate key. I found few blogs which says we can achieve this by explicit mention of schema. However, I have around 40 odd fiel...
- 648 Views
- 2 replies
- 1 kudos
- 10386 Views
- 3 replies
- 3 kudos
Unity Catalog Volume as spark checkpoint location
Hi,I tried to set the spark checkpoint location in a notebook to a folder in a Unity Catalog Volume, with the following command: sc.setCheckpointDir("/Volumes/catalog_name/schema_name/volume_name/folder_name")Unfortunately I receive the following err...
- 10386 Views
- 3 replies
- 3 kudos
- 3 kudos
Did you get any solution for the above issue? I am also trying same in DBR 15.4, Standard cluster .So I am able to set checkpoint directory using below commands. spark.conf.set("pyspark.sql.DataFrame.checkpoint", "/Volumes/path/") spark.conf.set("sp...
- 3 kudos
- 1009 Views
- 0 replies
- 1 kudos
Dedicated Access Mode Interactive Cluster with a Service Principal
Hi, I am wondering if it is possible to set up an interactive cluster set to dedicated access mode and having that user be a machine user?I've tried the cluster creation API, /api/2.1/clusters/create, and set the user name to the service principal na...
- 1009 Views
- 0 replies
- 1 kudos
- 48607 Views
- 44 replies
- 25 kudos
Resolved! Cannot sign in at databricks partner-academy portal
Hi thereI have used my company email to register an account for customer-academy.databricks.com a while back. Now what I need to do is create an account with partner-academy.databricks.com using my company email too.However when I register at partner...
- 48607 Views
- 44 replies
- 25 kudos
- 25 kudos
i'm having the same issue too, My company is a databricks partner and I registered to the customer portal before, and now i can't register to the Partner one. Could you please help me to migrate my account to Partner account?
- 25 kudos
- 1350 Views
- 3 replies
- 0 kudos
Connection timeout when connecting to MongoDB using MongoDB Connector for Spark 10.x
Hi.I'm testing a databricks connection to a mongo cluster V7 (azure cluster) using the library org.mongodb.spark:mongo-spark-connector_2.13:10.4.1I can connect using compass but I get a timeout error using my adb notebookMongoTimeoutException: Timed ...
- 1350 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi. Not a solution I'm afraid, but I'm having the exact same issue. Did you manage to resolve at all? What is throwing me is that I'm configuring the IP for the MongoDB instance as its running in AWS on an EC2 instance, but I still see 'localhost' on...
- 0 kudos
- 1459 Views
- 7 replies
- 0 kudos
Optimizing .collect() Usage in Spark
Hi all!I'm facing an issue with driver memory after deploying a cluster with 14GB of memory. My code utilizes the cluster’s compute power continuously (it never shuts down, as I cannot communicate with the Azure PostgreSQL database otherwise at the m...
- 1459 Views
- 7 replies
- 0 kudos
- 0 kudos
I would expect both the Python process on the driver and Spark's JVM to release memory once you are done with each chunk of data. Otherwise, this sounds like a memory leak. If you suspect this is a problem in the JVM, you can look at heap dumps - the...
- 0 kudos
- 7249 Views
- 7 replies
- 7 kudos
Run Delta Live Tables as service principal
How to run a delta live tables pipeline in production? It uses the owner's (creator's) permissions for writing to tables, and I can't change the owner of a UC-enabled pipeline after creation. I don't want regular users to have write access to prod ta...
- 7249 Views
- 7 replies
- 7 kudos
- 7 kudos
@knutasmI have build the solution for it. The way to create DLT pipeline using SPN is to write a code wherein via databricks API a new DLT pipeline will be created and you mentioned owner as a service principal in the API code as shown below. Below m...
- 7 kudos
- 980 Views
- 0 replies
- 0 kudos
Can you default to `execution-count: none` when stripping notebook outputs?
When committing to a git folder, IPYNB outputs are usually stripped, unless allowed by an admin setting and toggled by .databricks/commit_outputs. This sets the{"execution-count": 0, ... }within the IPYNB metadata. Is there a way to set it instead to...
- 980 Views
- 0 replies
- 0 kudos
- 11743 Views
- 4 replies
- 0 kudos
Data Lineage in Unity Catalog not Populating
I have been trying to get the data lineage to populate with the simplest of queries on a unity enabled catalog with a unity enabled cluster. I am essentially running the example provided with more data to see how it works with various aggregates dow...
- 11743 Views
- 4 replies
- 0 kudos
- 0 kudos
Cluster running in shared mode with Databricks runtime 15.4 LTSUC setup as per Databricks guide. I can see system tables everything but data is not populating in the table_lineage table. EventHub Firewall 9093 port is also open.Enabled runtime settin...
- 0 kudos
- 379 Views
- 1 replies
- 1 kudos
Resolved! Lab material for "Apache Spark Developer Learning Plan"
Hi, I just finished the following course "Introduction to Python for Data Science and Data Engineering". The instructor talks about the lab material but no where in the tabs on the left hand side, I am able to find any link to the lab material. I am ...
- 379 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @HeyRam! Did you take the Self-paced course? To clarify, lab materials are not available in self-paced courses. To access them, you have two options: Enroll in the ILT (Instructor-Led Training) course - This will grant you access to the labs fo...
- 1 kudos
- 503 Views
- 3 replies
- 0 kudos
removal of Delta live tables
Hello Team,I have removed definition of table from delta live table pipeline but table is still present in unity catalog. In event log, it is giving below messageMaterialized View '`catalog1`.`schema1`.`table1`' is no longer defined in the pipeline a...
- 503 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Anish_2 How are you doing today? I agree with @KaranamS's answer.Databricks marks the table as inactive instead of removing it to prevent accidental data loss, allowing you to restore it if needed. Once inactive, the table remains in Unity Catalo...
- 0 kudos
- 1161 Views
- 11 replies
- 4 kudos
DLT Pipeline event_log error - invalid pipeline name / The Spark SQL phase analysis failed
I am trying the below queries using both SQL warehouse and a shared cluster on Databricks runtime (15.4/16.1) with Unity Catalog: SELECT * FROM event_log(table(my_catalog.myschema.bronze_employees))SELECT * FROM event_log("6b317553-5c5a-40d5-9541-1a5...
- 1161 Views
- 11 replies
- 4 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
