cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

v_n66
by New Contributor
  • 438 Views
  • 7 replies
  • 1 kudos

dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

dbutils.notebooks.exit() is not returning results to the variable only on some notebooks issue is Parent child notebook communication and need some solution

  • 438 Views
  • 7 replies
  • 1 kudos
Latest Reply
lorenzoscandola
New Contributor II
  • 1 kudos

@SteveW atm everything works! 

  • 1 kudos
6 More Replies
cristianc
by Contributor
  • 651 Views
  • 2 replies
  • 1 kudos

Resolved! Does Databricks support AWS S3 Express One Zone?

Greetings,I'm writing this message since I learned that AWS has a storage class that is faster than S3 Standard called "S3 Express One Zone". (https://aws.amazon.com/s3/storage-classes/express-one-zone/)AWS offers support for this storage class with ...

  • 651 Views
  • 2 replies
  • 1 kudos
Latest Reply
Walter_C
Databricks Employee
  • 1 kudos

Right now there is no support for S3 Express One Zone but this is already in our radar through idea DB-I-8058, this is currently tagged as Considered for the future, there is no ETA but our teams are working to have this supported in the near future.

  • 1 kudos
1 More Replies
Phani1
by Valued Contributor II
  • 577 Views
  • 4 replies
  • 0 kudos

Triggering DLT Pipelines with Dynamic Parameters

Hi Team,We have a scenario where we need to pass a dynamic parameter to a Spark job that will trigger a DLT pipeline in append mode. Can you please suggest an approach for this?Regards,Phani

  • 577 Views
  • 4 replies
  • 0 kudos
Latest Reply
Phani1
Valued Contributor II
  • 0 kudos

I want to trigger a Delta Live Tables (DLT) pipeline from a Databricks Job and pass a dynamic input parameter to apply a filter. However, it seems that pipeline settings can only be defined when creating the pipeline, and not when executing it. Is th...

  • 0 kudos
3 More Replies
daan_dw
by New Contributor III
  • 80 Views
  • 2 replies
  • 0 kudos

Resolved! Auto Loader for copying files on s3

Hey community,I have a folder on s3 with around 5 million small files. On a daily basis new files are added. I would like to simply copy those new files to another folder on s3. My approach is to use an Auto Loader of which I attached the code below....

  • 80 Views
  • 2 replies
  • 0 kudos
Latest Reply
daan_dw
New Contributor III
  • 0 kudos

Hey LRALVAThe first time running your code I got the error:  PicklingError: Could not serialize object: Exception: You cannot use dbutils within a spark job You cannot use dbutils within a spark job or otherwise pickle it.So I changed the copy_single...

  • 0 kudos
1 More Replies
GJ2
by New Contributor II
  • 2561 Views
  • 7 replies
  • 1 kudos

Install the ODBC Driver 17 for SQL Server

Hi,I am not a Data Engineer, I want to connect to ssas. It looks like it can be connected through pyodbc. however looks like  I need to install "ODBC Driver 17 for SQL Server" using the following command. How do i install the driver on the cluster an...

GJ2_1-1739798450883.png
  • 2561 Views
  • 7 replies
  • 1 kudos
Latest Reply
karifclark
New Contributor
  • 1 kudos

You are correct that installing the "ODBC Driver 17 for SQL Server" is necessary in order to connect to SSAS using pyodbc. Using the proper system package manager (apt for Ubuntu or yum for CentOS), you may install it on your cluster. If it's a distr...

  • 1 kudos
6 More Replies
Phani1
by Valued Contributor II
  • 15 Views
  • 0 replies
  • 0 kudos

Genie Integrating with streamlit

Hi All,What are the best practices to follow while integrating with Genie and streamlit , and are there any limitations?how best way to present in UI level on user perceptive ?Regards,Phani

  • 15 Views
  • 0 replies
  • 0 kudos
skosaraju
by Visitor
  • 44 Views
  • 3 replies
  • 0 kudos

Getting an error "You cannot use dbutils within a spark job"

I am using mlflow to register my custom model using a simple code below. The DatabricksParams extracts all the params from dbutils and sets the params dictionary and dbutils is not used anywhere else within the rest of my code base. The code fails wh...

  • 44 Views
  • 3 replies
  • 0 kudos
Latest Reply
skosaraju
Visitor
  • 0 kudos

Ah.. I get it now. The params is an instance of the DatabricksParams which contains the dbutils..Let me try fixing this. Will let you know if this worked.

  • 0 kudos
2 More Replies
Nick_Pacey
by New Contributor III
  • 942 Views
  • 3 replies
  • 1 kudos

Issue when trying to create a Foreign Catalog to a On Prem SQL Server Instance

Hi,We are creating a lakehouse federated connection to our 2016 On Prem SQL Server.  This has an instance in place, so we only want and need to connect to this instance.  From this connection, we want to create a foreign catalog of a database on the ...

  • 942 Views
  • 3 replies
  • 1 kudos
Latest Reply
trueray_3150
New Contributor
  • 1 kudos

Hi @Nick_Pacey  Thank you I already did that using in the codejdbc_url = "jdbc:sqlserver://999.99.999.99\\instance:7777;encrypt=true;trustServerCertificate=true;database=mydatabase"jdbc_username = "myusername"jdbc_password = "mypassword"jdbc_driver =...

  • 1 kudos
2 More Replies
TamD
by Contributor
  • 26 Views
  • 0 replies
  • 0 kudos

Cannot apply liquid clustering via DLT pipeline

I want to use liquid clustering on a materialised view created via a DLT pipeline, however, there doesn't appear to be a valid way to do this.Via table properties:@Dlt.table( name="<table name>, comment="<table description", table_propert...

  • 26 Views
  • 0 replies
  • 0 kudos
jash281098
by New Contributor
  • 103 Views
  • 1 replies
  • 0 kudos

Issues when adding keystore spark config for pyspark to mongo atlas X.509 connectivity

Step followed - Step1: To add init script that will copy the keystore file in the tmp location.Step2: To add spark config in cluster advance options - spark.driver.extraJavaOptions -Djavax.net.ssl.keyStore=/tmp/keystore.jks -Djavax.net.ssl.keyStorePa...

  • 103 Views
  • 1 replies
  • 0 kudos
Latest Reply
jash281098
New Contributor
  • 0 kudos

Error Messages -Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Could not connect to address=(host=mdb7sywh50xhpr.chkweekm4xjq.us-east-1.rds.amazonaws.com)(port=3306)(type=master) : Could not conne...

  • 0 kudos
Direo
by Contributor II
  • 30104 Views
  • 6 replies
  • 1 kudos

Resolved! Importing CA certificate into a Databricks cluster

Hi!I was following guide outlined here:https://kb.databricks.com/en_US/python/import-custom-ca-cert(also tried this: https://stackoverflow.com/questions/73043589/configuring-tls-ca-on-databricks)to add ca root certificate into Databricks cluster, but...

  • 30104 Views
  • 6 replies
  • 1 kudos
Latest Reply
jash281098
New Contributor
  • 1 kudos

@Debayan One question - Will same approach work for JKS file containing private key certificate for X.509 authentication to Mongo Atlas database.Usual way of adding below spark config's is not working. spark.driver.extraJavaOptions -Djavax.net.ssl.ke...

  • 1 kudos
5 More Replies
DaPo
by New Contributor II
  • 615 Views
  • 1 replies
  • 0 kudos

DLT Fails with Exception: CANNOT_READ_STREAMING_STATE_FILE

I have several DLT Pipeline, writing to some schema in a unity catalog. The storage location of the unity-catalog is managed by the databricks deployment (on AWS).The schema and the dlt-pipeline are managed via databricks asset bundles. I did not cha...

  • 615 Views
  • 1 replies
  • 0 kudos
Latest Reply
mani_22
Databricks Employee
  • 0 kudos

Hi @DaPo , Have you made any code changes to your streaming query? There are limitations on what changes in a streaming query are allowed between restarts from the same checkpoint location. Refer this documentation The checkpoint location appears to ...

  • 0 kudos
Stentone
by New Contributor
  • 134 Views
  • 3 replies
  • 0 kudos

DLT Direct Publish Mode does not Handle Constraint Dependencies

I'm having some issues with the direct publish mode when defining a DLT workflow that includes tables where their schema defines foreign key constraints. When the foreign constraints reference tables that are not directly defined in any joins of the ...

  • 134 Views
  • 3 replies
  • 0 kudos
Latest Reply
LRALVA
Valued Contributor II
  • 0 kudos

@Stentone This is a tricky situation where you want to leverage the metadata benefits (like the ERD visualization) without running into execution dependencies. Let me help you solve this issue.The error suggests that DLT is trying to validate the for...

  • 0 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels