- 438 Views
- 7 replies
- 1 kudos
dbutils.notebooks.exit() is not returning results to the variable only on some notebooks
dbutils.notebooks.exit() is not returning results to the variable only on some notebooks issue is Parent child notebook communication and need some solution
- 438 Views
- 7 replies
- 1 kudos
- 651 Views
- 2 replies
- 1 kudos
Resolved! Does Databricks support AWS S3 Express One Zone?
Greetings,I'm writing this message since I learned that AWS has a storage class that is faster than S3 Standard called "S3 Express One Zone". (https://aws.amazon.com/s3/storage-classes/express-one-zone/)AWS offers support for this storage class with ...
- 651 Views
- 2 replies
- 1 kudos
- 1 kudos
Right now there is no support for S3 Express One Zone but this is already in our radar through idea DB-I-8058, this is currently tagged as Considered for the future, there is no ETA but our teams are working to have this supported in the near future.
- 1 kudos
- 577 Views
- 4 replies
- 0 kudos
Triggering DLT Pipelines with Dynamic Parameters
Hi Team,We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?Regards,Phani
- 577 Views
- 4 replies
- 0 kudos
- 0 kudos
I want to trigger a Delta Live Tables (DLT) pipeline from a Databricks Job and pass a dynamic input parameter to apply a filter. However, it seems that pipeline settings can only be defined when creating the pipeline, and not when executing it. Is th...
- 0 kudos
- 80 Views
- 2 replies
- 0 kudos
Resolved! Auto Loader for copying files on s3
Hey community,I have a folder on s3 with around 5 million small files. On a daily basis new files are added. I would like to simply copy those new files to another folder on s3. My approach is to use an Auto Loader of which I attached the code below....
- 80 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey LRALVAThe first time running your code I got the error: PicklingError: Could not serialize object: Exception: You cannot use dbutils within a spark job You cannot use dbutils within a spark job or otherwise pickle it.So I changed the copy_single...
- 0 kudos
- 20 Views
- 0 replies
- 0 kudos
Not able to read sample data in databricks in shared cluster but using single cluster im able to
Im not able to view sample data using share clusterError getting sample datasocket closedBut when I use the single cluster mode Im able to read the data
- 20 Views
- 0 replies
- 0 kudos
- 11 Views
- 0 replies
- 0 kudos
Schema Evolution with "schemaTrackingLocation" fails anyway
Hi, I'm trying to understand the usage of "schemaTrackLocation" with schema evolution.I use these articles as references:https://docs.delta.io/latest/delta-streaming.html#tracking-non-additive-schema-changeshttps://docs.databricks.com/aws/en/error-me...
- 11 Views
- 0 replies
- 0 kudos
- 2561 Views
- 7 replies
- 1 kudos
Install the ODBC Driver 17 for SQL Server
Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...
- 2561 Views
- 7 replies
- 1 kudos
- 1 kudos
You are correct that installing the "ODBC Driver 17 for SQL Server" is necessary in order to connect to SSAS using pyodbc. Using the proper system package manager (apt for Ubuntu or yum for CentOS), you may install it on your cluster. If it's a distr...
- 1 kudos
- 15 Views
- 0 replies
- 0 kudos
Genie Integrating with streamlit
Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani
- 15 Views
- 0 replies
- 0 kudos
- 44 Views
- 3 replies
- 0 kudos
Getting an error "You cannot use dbutils within a spark job"
I am using mlflow to register my custom model using a simple code below. The DatabricksParams extracts all the params from dbutils and sets the params dictionary and dbutils is not used anywhere else within the rest of my code base. The code fails wh...
- 44 Views
- 3 replies
- 0 kudos
- 0 kudos
Ah.. I get it now. The params is an instance of the DatabricksParams which contains the dbutils..Let me try fixing this. Will let you know if this worked.
- 0 kudos
- 942 Views
- 3 replies
- 1 kudos
Issue when trying to create a Foreign Catalog to a On Prem SQL Server Instance
Hi,We are creating a lakehouse federated connection to our 2016 On Prem SQL Server. This has an instance in place, so we only want and need to connect to this instance. From this connection, we want to create a foreign catalog of a database on the ...
- 942 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nick_Pacey Thank you I already did that using in the codejdbc_url = "jdbc:sqlserver://999.99.999.99\\instance:7777;encrypt=true;trustServerCertificate=true;database=mydatabase"jdbc_username = "myusername"jdbc_password = "mypassword"jdbc_driver =...
- 1 kudos
- 26 Views
- 0 replies
- 0 kudos
Cannot apply liquid clustering via DLT pipeline
I want to use liquid clustering on a materialised view created via a DLT pipeline, however, there doesn't appear to be a valid way to do this.Via table properties:@Dlt.table( name="<table name>, comment="<table description", table_propert...
- 26 Views
- 0 replies
- 0 kudos
- 103 Views
- 1 replies
- 0 kudos
Issues when adding keystore spark config for pyspark to mongo atlas X.509 connectivity
Step followed - Step1: To add init script that will copy the keystore file in the tmp location.Step2: To add spark config in cluster advance options - spark.driver.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePa...
- 103 Views
- 1 replies
- 0 kudos
- 0 kudos
Error Messages -Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Could not connect to address=(host=mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : Could not conne...
- 0 kudos
- 30104 Views
- 6 replies
- 1 kudos
Resolved! Importing CA certificate into a Databricks cluster
Hi!I was following guide outlined here:https://kb.databricks.com/en_US/python/import-custom-ca-cert(also tried this: https://stackoverflow.com/questions/73043589/configuring-tls-ca-on-databricks)to add ca root certificate into Databricks cluster, but...
- 30104 Views
- 6 replies
- 1 kudos
- 1 kudos
@Debayan One question - Will same approach work for JKS file containing private key certificate for X.509 authentication to Mongo Atlas database.Usual way of adding below spark config's is not working. spark.driver.extraJavaOptions -Djavax.net.ssl.ke...
- 1 kudos
- 615 Views
- 1 replies
- 0 kudos
DLT Fails with Exception: CANNOT_READ_STREAMING_STATE_FILE
I have several DLT Pipeline, writing to some schema in a unity catalog. The storage location of the unity-catalog is managed by the databricks deployment (on AWS).The schema and the dlt-pipeline are managed via databricks asset bundles. I did not cha...
- 615 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DaPo , Have you made any code changes to your streaming query? There are limitations on what changes in a streaming query are allowed between restarts from the same checkpoint location. Refer this documentation The checkpoint location appears to ...
- 0 kudos
- 134 Views
- 3 replies
- 0 kudos
DLT Direct Publish Mode does not Handle Constraint Dependencies
I'm having some issues with the direct publish mode when defining a DLT workflow that includes tables where their schema defines foreign key constraints. When the foreign constraints reference tables that are not directly defined in any joins of the ...
- 134 Views
- 3 replies
- 0 kudos
- 0 kudos
@Stentone This is a tricky situation where you want to leverage the metadata benefits (like the ERD visualization) without running into execution dependencies. Let me help you solve this issue.The error suggests that DLT is trying to validate the for...
- 0 kudos
| User | Count |
|---|---|
| 1611 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
