- 403 Views
- 1 replies
- 0 kudos
Resolved! Databricks ro Salesforce | Unity Catalog Query
AskCan we get a UC catalog (like prod or genie) in free edition of data bricks ?Problem i am solving:Structuring Data in Databricks before sending customer, account data to salesforceissue:cannot see workspace-local tables (workspace.default.structur...
- 403 Views
- 1 replies
- 0 kudos
- 0 kudos
root cause of not seeing your workspace-local tables (workspace.default.structured_pdf_table) is the unavailability of a Unity Catalog or Delta Sharing connector configuration in your Free Edition workspace. To resolve this, you typically need admin ...
- 0 kudos
- 1063 Views
- 4 replies
- 0 kudos
`io.unitycatalog.client.model.TableType`, Unexpected value 'MATERIALIZED_VIEW
I have been able to set up jdbc driver with databricks to connect to my unity catalog using local spark sessions. When i try to retrieve tables in my schema i get this error An error occurred while calling o43.sql.: io.unitycatalog.client.ApiExcepti...
- 1063 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @NUKSY , @Jofes This should be reported as a bug, see similar issues reported. https://github.com/unitycatalog/unitycatalog/issues/657 https://github.com/unitycatalog/unitycatalog/issues/1077 Thanks!
- 0 kudos
- 590 Views
- 2 replies
- 2 kudos
Resolved! delta live tables
Hi . i have a source table that is a delta live streaming table created using dlt.auto_cdc logic and now i want to create another sreaming table that filters the record from that table as per the client but it also should have auto cdc logic for the...
- 590 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @tenzinpro , This is an expected error. "DELTA_SOURCE_TABLE_IGNORE_CHANGES] Detected a data update" As explained in the error: This is currently not supported. If this is going to happen regularly and you are okay to skip changes, set the option ...
- 2 kudos
- 422 Views
- 3 replies
- 0 kudos
Embed AI/BI Dashboards into Databricks App
Hello,I’m interested in understanding whether it’s possible to embed multiple AI/BI dashboards created in Databricks within a Databricks app. Could you please share the steps or provide any documentation related to this? My goal is to use the app as ...
- 422 Views
- 3 replies
- 0 kudos
- 0 kudos
Just checking if anyone has already implemented this. Pls share your thoughts.
- 0 kudos
- 417 Views
- 4 replies
- 1 kudos
How to fetch spark.addFiles when used multi node cluster
I wanted to share the nearly 12 xml files from dbfs location to executor local path by using sc.addFile and I went to your blog and tweaked my code to form path with file:/// the result of it was it worked when we have only one node but throwed erro...
- 417 Views
- 4 replies
- 1 kudos
- 1 kudos
Hello @Mahesh_rathi__ , SparkContext.addFile is for shipping small side files to executors, not for creating an input path that you can pass to sc.textFile("file://..."). On a single-node cluster the driver and executor share the same machine, so the...
- 1 kudos
- 914 Views
- 9 replies
- 13 kudos
Workflows now harder to find old failed runs
Some time in the past few weeks I think there was an update to databricks workflows. Previously you could:run a workflowit failsrepair the workflowclick into the workflowview past runs before that failed via a dropdown bar (like in the screenshot bel...
- 914 Views
- 9 replies
- 13 kudos
- 13 kudos
I am a Databricks Admin in the workspace that @kenmyers-8451 is having problems in and I am getting the same issue where the retries show up for a brief second but then redirect/refresh and then the retries disappear.This seems to happen when the wor...
- 13 kudos
- 286 Views
- 2 replies
- 0 kudos
Serverless base env setup in Databricks Asset Bundle (DAB)
I am trying to set a base environment for my task (notebook) which is running on serverless, following is the dab yaml i am using when i did bundle deploy -t users, its not throwing any error but its not installing the libraries from the base env, ca...
- 286 Views
- 2 replies
- 0 kudos
- 0 kudos
Your YAML is valid, but the reason the libraries are not being installed is because base_environment_path is not supported for serverless compute. Serverless jobs use a fully managed environment and you can’t override it with a custom base environmen...
- 0 kudos
- 411 Views
- 8 replies
- 1 kudos
JDBC driver CPU consumption
Hi,I am using JDBC driver to execute an insert statement with several thousand of rows (~4MB). It takes several seconds to complete and for some reason consumes 1 full CPU core for it.It seems like a lot of the time is spent in this method:com.databr...
- 411 Views
- 8 replies
- 1 kudos
- 1 kudos
Hi @ivni ,Yes, that method could be CPU intensive. According to driver's docs it removes catalog name from query statement. But it doing this via regex patterns - this is heavy operation from CPU perspective, especially if you have a lot of complex q...
- 1 kudos
- 370 Views
- 1 replies
- 1 kudos
Issue with Lakebridge transpile installation – SSL Certificate Error
Hi Team,I am trying to use Lakebridge to test a small piece of code for conversion. The base installation of Lakebridge worked fine, but when I attempt to install transpile, I encounter SSL-related errors. I even tried to hardcode of the certificates...
- 370 Views
- 1 replies
- 1 kudos
- 1 kudos
The error indicates that while installing Lakebridge's transpile component on Databricks with Python 3.13, SSL certificate verification fails due to a "Missing Authority Key Identifier" in the certificate chain. This is a result of stricter requireme...
- 1 kudos
- 384 Views
- 3 replies
- 2 kudos
Resolved! Inconsistent behavior of LakeBridge transpiler for similar scripts
Hi Everyone,I am testing the LakeBridge prototype and noticed inconsistent behavior when converting stored procedures.For simple scripts, the conversion is correct.But for medium/complex scripts, especially those with multiple LEFT JOINs and column e...
- 384 Views
- 3 replies
- 2 kudos
- 2 kudos
Is there a way to standardize/optimize the conversion so it doesn’t blow up into thousands of lines? Yes, there are actionable methods to standardize and optimize LakeBridge conversions to prevent code from ballooning into thousands of lines, but t...
- 2 kudos
- 290 Views
- 1 replies
- 2 kudos
Resolved! Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor
Hello,I run my process on a shared, interactive cluster (data security mode: USER_ISOLATION).I run operations on multiple tables having each of them as a separate thread, pseudo-code :try: truncate target_tablevacuum target_table (retain 0 hours with...
- 290 Views
- 1 replies
- 2 kudos
- 2 kudos
This is most likely a bug. It certainly is unexpected and should be reported to Databricks support or your platform administrator for clarification and remediation. As a temporary workaround, running vacuum commands outside of threaded contexts or s...
- 2 kudos
- 3709 Views
- 5 replies
- 0 kudos
Delta Live tables stream output to Kafka
Hello,Wanted to Know if we can write the stream output to a Kafka topic in DLT pipeline?Please let me know.Thankyou.
- 3709 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi ! Ensure your code is set up to use these libraries. Here is the complete example: Navigate to your cluster configuration:Go to your Databricks workspace.Click on "Clusters" and select your cluster.Go to the "Libraries" tab. Install the necessar...
- 0 kudos
- 392 Views
- 5 replies
- 0 kudos
Databricks bundle repository permission
Hi everyone,How can I use Databricks Asset Bundle configuration to set permissions on the workspace folder (root_path) where my code is deployed, in order to protect it from manual changes by users?My current bundle config for production looks like t...
- 392 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @HoussemBL ,In Databricks there is a users group to which by deafult all the workspace users belong (in UI displayed All workspace users). That group has default permission that cannot be revoked at the top-level Shared folder. So, any new folder ...
- 0 kudos
- 20262 Views
- 25 replies
- 28 kudos
Installing Databricks Connect breaks pyspark local cluster mode
Hi, It seems that when databricks-connect is installed, pyspark is at the same time modified so that it will not anymore work with local master node. This has been especially useful in testing, when unit tests for spark-related code without any remot...
- 20262 Views
- 25 replies
- 28 kudos
- 28 kudos
Hi Utu,try doing something like this , wrap import within a fixture itself.import osimport pytestfrom pyspark.sql import SparkSession_local_test = True@pytest.fixture(scope='session')def spark(): if 'DATABRICKS_RUNTIME_VERSION' in os.environ: ...
- 28 kudos
- 1191 Views
- 6 replies
- 7 kudos
Resolved! Retrieving Last Data and Metadata Refresh DateTimes of table in Databricks
Hi, I had a query regarding how to accurately retrieve the last data refresh datetime and last metadata refresh datetime for tables in Databricks. Currently, the only reliable approach I am aware of is using the DESCRIBE HISTORY command with filters ...
- 1191 Views
- 6 replies
- 7 kudos
| User | Count |
|---|---|
| 1616 | |
| 789 | |
| 442 | |
| 349 | |
| 287 |
