cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Yoni
by New Contributor
  • 15088 Views
  • 5 replies
  • 3 kudos

Resolved! MLFlow failed: You haven't configured the CLI yet

I'm getting an errorYou haven’t configured the CLI yet! Please configure by entering `/databricks/python_shell/scripts/db_ipykernel_launcher.py configure`My cluster is running Databricks Runtime Version 10.1I've also installed mlflow to the cluster l...

  • 15088 Views
  • 5 replies
  • 3 kudos
Latest Reply
HemantKumar
New Contributor II
  • 3 kudos

dbutils.library.restartPython()Add that after you run the pip install mlflow, it worked for me in a non-ML cluster

  • 3 kudos
4 More Replies
DumbBeaver
by New Contributor II
  • 2014 Views
  • 2 replies
  • 1 kudos

Resolved! ERROR: Writing to Unity Catalog from Remote Spark using JDBC

This is my code here. df = spark.createDataFrame([[1,1,2]], schema=['id','first_name','last_name'])(df.write.format("jdbc")     .option("url",  <jdbc-url>)    .option("dbtable","hive_metastore.default.test")    .option("driver", "com.databricks.clien...

  • 2014 Views
  • 2 replies
  • 1 kudos
Latest Reply
feiyun0112
Honored Contributor
  • 1 kudos

 %scala import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects} JdbcDialects.registerDialect(new JdbcDialect() { override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:databricks:") override def quoteIde...

  • 1 kudos
1 More Replies
Azure_Data_Bric
by New Contributor III
  • 3084 Views
  • 6 replies
  • 0 kudos

Historical Data Clean-up from Silver tables

Hi Everyone,I need your help/suggestion.We are using a DLT framework for our ELT process, data is received from the Source to the RAW layer in parquet format. This raw data is loaded to the Bronze layer which acts like a history table. From the BRONZ...

  • 3084 Views
  • 6 replies
  • 0 kudos
Latest Reply
Azure_Data_Bric
New Contributor III
  • 0 kudos

Hi,I see Optimize and VACUUM are running on all tables once per day automatically.that day when we performed historical deletion, we deleted the data first, and then we just ran VACUUM with zero hour retention. After some time Optimize and VACUUM (wi...

  • 0 kudos
5 More Replies
CloudPlatformer
by New Contributor II
  • 3049 Views
  • 1 replies
  • 0 kudos

Npip Tunnel Setup Failure

Hi everyone,I'm currently running into an issue when trying to create any type of compute cluster in a workspace (premium, with VNet Injection and private DNS zone + private Endpoint). The operation always fails with: Compute terminated. Reason: Npip...

  • 3049 Views
  • 1 replies
  • 0 kudos
Latest Reply
CloudPlatformer
New Contributor II
  • 0 kudos

I forgot to add: the workspace as well as the other resources are hosted in Azure.

  • 0 kudos
Etyr
by Contributor
  • 1680 Views
  • 2 replies
  • 0 kudos

Can not connect to databricks on Azure Machine Learning Compute Cluster.

Hello,I'am having an issue where I have :A local machine in WSL 1,Python 3.8 and 3.10OpenJDK 19.0.1 (version "build 19.0.1+10-21")Compute Instance In Azure Machine LearningPython 3.8OpenJDK 8 (version "1.8.0_392")Compute Cluster in Azure Machine Lear...

  • 1680 Views
  • 2 replies
  • 0 kudos
Latest Reply
Etyr
Contributor
  • 0 kudos

Additional information I forgot to write.Compute Instance has a User managed Identity in Azure, a Service Principal access is created in databricks with its Application ID. Same with the compute cluster, it has its own User Managed Identity that is a...

  • 0 kudos
1 More Replies
Mado
by Valued Contributor II
  • 12479 Views
  • 3 replies
  • 0 kudos

Resolved! How to enforce delta table column to have unique values?

Hi,I have defined a delta table with a primary key:%sql   CREATE TABLE IF NOT EXISTS test_table_pk ( table_name STRING NOT NULL, label STRING NOT NULL, table_location STRING NOT NULL,   CONSTRAINT test_table_pk_col PRIMARY KEY(table_name) ...

image
  • 12479 Views
  • 3 replies
  • 0 kudos
Latest Reply
Steve_Lyle_BPCS
New Contributor II
  • 0 kudos

I'm with you.  But it DOES make sense because DBx databases are not application databases.  DBx is not intended to be used like this.  DBx databases are repositories for any ingested abstract data.  To manage the ingestion is purpose-built databases ...

  • 0 kudos
2 More Replies
learning_1989
by New Contributor II
  • 2194 Views
  • 2 replies
  • 1 kudos

You have json file which is nested with multiple key value pair how you read it in databricks?

You have json file which is nested with multiple key value pair how you read it in databricks?

  • 2194 Views
  • 2 replies
  • 1 kudos
Latest Reply
Lakshay
Databricks Employee
  • 1 kudos

You should be able to read the json file with below code. val df = spark.read.format("json").load("file.json") After this you will need to use the explode function to add columns to the dataframe using the nested values.

  • 1 kudos
1 More Replies
RKNutalapati
by Valued Contributor
  • 2373 Views
  • 3 replies
  • 0 kudos

How to use Oracle Wallet to connect from databricks

How to onnect Databricks to Oracle DAS / Autonomous Database using a cloud wallet, What are the typical steps and best practices to follow. Appreciate an example code snippet for connecting to the above data source

  • 2373 Views
  • 3 replies
  • 0 kudos
Latest Reply
RKNutalapati
Valued Contributor
  • 0 kudos

 Followed below steps to build the connection.Unzip Oracle Wallet objects and copy them to a secure location accessible by your Databricks workspace.Collaborate with your Network team and Oracle Autonomous Instance Admins to open firewalls between yo...

  • 0 kudos
2 More Replies
Snentley
by New Contributor
  • 1484 Views
  • 1 replies
  • 0 kudos

Free Voucher for Data Engineering Associate Certification

Could you please inform me which specific webinar participation might grant eligibility for a certification exam voucher? Additionally, I would like to know whether this voucher would cover the full cost of the certification exam or only a partial am...

  • 1484 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kiv9
New Contributor II
  • 0 kudos

Did you get any response on this  

  • 0 kudos
Phani1
by Valued Contributor II
  • 632 Views
  • 1 replies
  • 0 kudos

Databricks masking

Should we convert the Python-based masking logic to SQL in databricks for implementing masking? Will the masking feature continue to work while connected to Power BI?Regards,Phanindra

  • 632 Views
  • 1 replies
  • 0 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 0 kudos

@Phani1 - could you please be more precise on the question. Are you discussing about mask  function in DBSQL?

  • 0 kudos
amama
by New Contributor II
  • 2272 Views
  • 3 replies
  • 1 kudos

How to run spark sql file through Azure Databricks

We have a process that will write spark sql to a file, this process will generate thousands of spark sql files in the production environment.These files will be created in the ADLS Gen2 directory.sample spark file---val 2023_I = spark.sql("select rm....

  • 2272 Views
  • 3 replies
  • 1 kudos
Latest Reply
shan_chandra
Databricks Employee
  • 1 kudos

@amama - you can mount the ADLS storage location in databricks. Since, this is a scala code, you can use workflow and create tasks to execute these scala code by providing the input as the mount location. 

  • 1 kudos
2 More Replies
marcusmv
by New Contributor II
  • 2315 Views
  • 2 replies
  • 1 kudos

Resolved! Advanced Data Engineering with Databricks course

I'm looking for materials to prepare for the Databricks Certified Professional Data Engineer exam. But I see two courses titled 'Advanced Data Engineering with Databricks' in the academy (E-VDG8QV andE-19WXD1). Which one of these courses should I be ...

Data Engineering
associate
exam
learning
professional
  • 2315 Views
  • 2 replies
  • 1 kudos
Latest Reply
marcusmv
New Contributor II
  • 1 kudos

Does anyone know? Would much appreciate it.

  • 1 kudos
1 More Replies
vpaluch
by New Contributor II
  • 3847 Views
  • 1 replies
  • 0 kudos

External Table from partitioned CSV in Unity Catalog.

When I create an External Table in unity catalog from a flattened csv folder, it  works as expected:     CREATE EXTERNAL LOCATION IF NOT EXISTS raw_data URL 'abfss://raw@storage0account0name.dfs.core.windows.net' WITH ( STORAGE CREDENTIAL `a579a...

Data Engineering
Partitioned_CSV
  • 3847 Views
  • 1 replies
  • 0 kudos
Latest Reply
vpaluch
New Contributor II
  • 0 kudos

Thanks Kaniz,I'm using an External Location authenticated using a Managed Identity. The very same used for the non-partitioned table and many others that works pretty fine. This account has Storage Blob Contributor rights for all containers and folde...

  • 0 kudos
Etyr
by Contributor
  • 5032 Views
  • 3 replies
  • 1 kudos

databricks.sql.exc.RequestError OpenSession error None

I'm trying to access to a Databricks SQL Warehouse with python. I'm able to connect with a token on a Compute Instance on Azure Machine Learning. It's a VM with conda installed, I create an env in python 3.10.from databricks import sql as dbsql dbsq...

  • 5032 Views
  • 3 replies
  • 1 kudos
Latest Reply
Etyr
Contributor
  • 1 kudos

The issue was that the new version of databricks-sql-connector (3.0.1) does not handle well error messages. So It gave a generic error and a timeout where it should have given me 403 and instant error message without a 900 second timeout.https://gith...

  • 1 kudos
2 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels