- 926 Views
- 4 replies
- 0 kudos
Error: from databricks import lakehouse_monitoring
We are using the following import: "from databricks import lakehouse_monitoring". We are receiving this error:ImportError: cannot import name 'lakehouse_monitoring' from 'databricks.sdk' (/databricks/python/lib/python3.11/site-packages/databricks/sdk...
- 926 Views
- 4 replies
- 0 kudos
- 0 kudos
I wasnt able to find that module. However there are options under sdk. Refer to Lakehouse monitoring SDK reference %python import databricks print(dir(databricks.sdk))
- 0 kudos
- 810 Views
- 1 replies
- 1 kudos
Lakeflow access
Hi,Can someone please advise how to sign up for Lakeflow access? I believe it's on public preview now, but it's not listed in my workspace's preview features list.Thanks!
- 810 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Fikrat, LakeFlow is currently in a gated Public Preview. To participate in the preview, you need to contact your Databricks account team. It is not listed in the workspace's preview features list because it requires specific access permissions th...
- 1 kudos
- 1196 Views
- 2 replies
- 0 kudos
How to Get the Size of Filtered Rows in Databricks SQL
I have a query that filters rows from a table based on a timestamp range. The query is as follows:SELECT COUNT(*) FROM table_name WHERE ts >= '2025-02-04 00:00:00' AND ts < '2025-02-05 00:00:00';This query returns 10 rows. I need to calculate the tot...
- 1196 Views
- 2 replies
- 0 kudos
- 0 kudos
@vidya_kothavale try this code block. Keep in mind to handle the null values.SELECT SUM(OCTET_LENGTH(CAST(column1 AS STRING)) + OCTET_LENGTH(CAST(column2 AS STRING)) + OCTET_LENGTH(CAST(COALESCE(column3, '0') AS STRING))) as bytes, SUM(OCTET...
- 0 kudos
- 1031 Views
- 2 replies
- 1 kudos
Urgent: Iceberg REST catalog - load a table has new JSON format
Hello, we are using Databricks Unity catalog to load an Iceberg table, i.e., Delta Lake table with Uniform feature). We are using this guide: https://docs.databricks.com/en/external-access/iceberg.html. This has been working for us since last year bu...
- 1031 Views
- 2 replies
- 1 kudos
- 1 kudos
Satyadeepak - it looks like this has been fixed on your end and we are no longer seeing this issue. Thanks for checking.
- 1 kudos
- 15843 Views
- 5 replies
- 1 kudos
DeltaFileNotFoundException: No file found in the directory (sudden task failure)
Hi all,I am currently running a job that will upsert a table by reading from delta change data feed from my silver table. Here is the relevent snippet of code: rds_changes = spark.read.format("delta") \ .option("readChangeFeed", "true") \ .optio...
- 15843 Views
- 5 replies
- 1 kudos
- 856 Views
- 1 replies
- 0 kudos
Autoloader: Cross-account bucket Assume role access denied
Hi everyone!I have a Databricks instance profile role that has permission to assume a role in another AWS account to access an S3 bucket in that account.When I try to assume the role using boto3, it correctly reads the Databricks AWS credentials, as...
- 856 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @deng_dev , Greetings!In the above error message, you will see a request ID in it, so can you please share that Request ID with the AWS Team to check why this request is getting denied as this looks like a permission issue. Please let me know if ...
- 0 kudos
- 1049 Views
- 3 replies
- 0 kudos
how to pull a parameter from .sql file with dbutils.notebook.run
Hi,I want to use this:result = dbutils.notebook.run('/Workspace/Usersxxxxt', 600, {"environment": inputEnvironment}) this pulls from this .sql file in that path:DROP TEMPORARY VARIABLE IF EXISTS strEnv;DECLARE VARIABLE strEnv STRING;SET VARIABLE strE...
- 1049 Views
- 3 replies
- 0 kudos
- 0 kudos
@carlos_tasayco There are two methods on how you can pass a variable to the other notebooks as input.Using Widgetsusing collect method like below. # In notebook1 result = spark.sql("SELECT value FROM table").collect()[0][0] dbutils.notebook.exit(resu...
- 0 kudos
- 3420 Views
- 4 replies
- 0 kudos
Resolved! What is the behaviour of starting version with spark structured streaming ?
Looking into the followinghttps://docs.databricks.com/en/structured-streaming/delta-lake.html#specify-initial-positionI am unclear as to what is the exact difference (if any) between "startingVersion: The Delta Lake version to start from. Databricks ...
- 3420 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @dlorenzo, interesting take! I don’t agree with your statement, though. According to both the documentation and my own testing, startingVersion = "latest" explicitly skips all historical data and starts from the latest committed version at the tim...
- 0 kudos
- 2051 Views
- 10 replies
- 0 kudos
Allows to serveless compute to connect to postgres db
Hi Community,Is it possible to enable VNet peering between Databricks Serverless Compute and a private PostgreSQL database that is already configured with a VNet?Currently, everything works fine when I create my personal cluster because I have set up...
- 2051 Views
- 10 replies
- 0 kudos
- 0 kudos
Is that PostgreSQL server going to go away after you migrate to Databricks, or is it going to continue to be used? Either way, federation works for you. If you're going to discontinue it, just do a full extract into an archive location and a one-ti...
- 0 kudos
- 5317 Views
- 3 replies
- 2 kudos
Resolved! [Simba][SparkJDBCDriver](500177) Error getting http path from connection string
I'm trying to use a very basic java program to connect to Databricks using spark jdbc driver (SparkJDBC42.jar), but I get the error (mentioned above): [Simba][SparkJDBCDriver](500177) Error getting http path from connection stringHere is my code snip...
- 5317 Views
- 3 replies
- 2 kudos
- 2 kudos
Hello @Claude Repono​ Thank you for posting your question in the community. It seems you were able to find the solution by yourself. That's awesome. We are going to go ahead and mark your answer as the best solution.
- 2 kudos
- 1086 Views
- 2 replies
- 0 kudos
system.access.table_lineage - source and target table meanings
I've been using the system.access.table_lineage table, and I'm trying to understand when the source and target tables are defined.For example, picking a specific job run and looking at the lineage:selectsource_type, source_table_full_name, target_typ...
- 1086 Views
- 2 replies
- 0 kudos
- 0 kudos
@Sidhant07thanks for the answer, I think it is good, but I am questioning scenario #5 (source=table,target=view).I'm looking at some examples in our table_lineage, and we aren't modifying the view or creating the view from within a job. I think scena...
- 0 kudos
- 6151 Views
- 5 replies
- 3 kudos
Resolved! databricks-connect 11.3
Would there be databricks-connect for cluster version 11.3 ?If yes, when we should expect it?
- 6151 Views
- 5 replies
- 3 kudos
- 3 kudos
It looks like there are other issues. I saved the model generated with the code above in mlflowWhen I try to reload it with this code:import mlflow model = mlflow.spark.load_model('runs:/cb6ff62587a0404cabeadd47e4c9408a/model') It works in a notebook...
- 3 kudos
- 882 Views
- 1 replies
- 1 kudos
DLT - Handling Merge
Hi,In our DLT pipeline we are reading two tables. One a Apply Changes table Delta table and a streaming live table. We are able to read the latest records from the streaming live table incrementally but from the apply changes we are not able to read ...
- 882 Views
- 1 replies
- 1 kudos
- 1 kudos
To address the challenges you are facing with your Delta Live Tables (DLT) pipeline, here are some steps and considerations to help you manage the incremental data reading and joining of the Apply Changes table and the streaming live table for SCD Ty...
- 1 kudos
- 883 Views
- 2 replies
- 0 kudos
Upgrading runtime 10.4 to 11.3 causing errors in my code (CASTING issues?)
Hi all,We have our medallion architecture transformation on databricks.Im currently testing upgrading to 11.3 as 10.4 won't be supported anymore from March 2025.However, I keep getting errors like this:Error inserting data into table. Type AnalysisEx...
- 883 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Alberto_Umana Thank you for your response.That's the weird thing. The RawDataStartDate only consists of records with datetime stamps. Furthermore I am nowhere in my code casting anything of this to a boolean, or casting anything at all. All I am ...
- 0 kudos
- 1892 Views
- 7 replies
- 2 kudos
Resolved! data bricks community edition login issue
I have a data bricks community edition account. I know the username and password. I used this account long time ago. I try to login this account, it is sent a verification code to my mail id. But I am unable to login my Gmail account doe to I forgot ...
- 1892 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi Im having an issue while logging into the databricks community edition. Where in even if I give correct email address and OTP it says "We were not able to find a Community Edition workspace with this email."could you please help?
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 493 | |
| 349 | |
| 287 |
