cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 
Data + AI Summit 2024 - Data Engineering & Streaming

Forum Posts

Brad
by Contributor
  • 1607 Views
  • 1 replies
  • 0 kudos

Why "rror: Invalid access to Org: xxx"

Hi team, I installed Databricks CLI, and run "databricks auth login --profile xxx" successfully. I can also connect from vscode to Databricks. "databricks clusters list -p xxx" also works. But when I tried to rundatabricks bundle validateI got"Error:...

  • 1607 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Brad,  Ensure that your Databricks CLI configuration is correctly set up with the right access token. Verify that the token is mentioned in both the password field and the Extra field.The Extra field should be configured with a JSON string like t...

  • 0 kudos
Ajay-Pandey
by Esteemed Contributor III
  • 1749 Views
  • 5 replies
  • 0 kudos

On-behalf-of token creation for service principals is not enabled for this workspace

Hi AllI just wanted to create PAT for Databricks Service Principle but getting below code while hitting API or using CLI - Please help me to create PAT for the same.#dataengineering #databricks

AjayPandey_0-1710845262519.png AjayPandey_1-1710845276557.png
Data Engineering
community
Databricks
  • 1749 Views
  • 5 replies
  • 0 kudos
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 0 kudos

Hi @Kaniz_Fatma Have you got any update on this ?

  • 0 kudos
4 More Replies
RajeshRK
by Contributor
  • 6350 Views
  • 6 replies
  • 0 kudos

Resolved! Need help to analyze databricks logs for a long-running job.

Hi Team,We have a job it completes in 3 minutes in one Databricks cluster, if we run the same job in another databricks cluster it is taking 3 hours to complete.I am quite new to Databricks and need your guidance on how to find out where databricks s...

  • 6350 Views
  • 6 replies
  • 0 kudos
Latest Reply
AmitKP
New Contributor II
  • 0 kudos

Hi @Kaniz_Fatma ,I am saving logs of my databricks Job Compute From ADF, How can i open those files that present in dbfs location.

  • 0 kudos
5 More Replies
prasad95
by New Contributor III
  • 658 Views
  • 2 replies
  • 0 kudos

How to unwrap the notebook code lines, By default its getting wrapping the code lines

How to unwrap the notebook code lines, By default its getting wrapping the code lines

  • 658 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @prasad95 , To prevent automatic line wrapping in Databricks Notebooks, follow these steps: Run Selected Text: If you want to execute specific code lines without wrapping, select the desired text and either: Click Run > Run selected text from ...

  • 0 kudos
1 More Replies
Gilg
by Contributor II
  • 1983 Views
  • 2 replies
  • 0 kudos

Move files

HiI am using DLT with Autoloader.DLT pipeline is running in Continuous mode.Autoloader is in Directory Listing mode (Default)Question.I want to move files that has been processed by the DLT to another folder (archived) and planning to have another no...

  • 1983 Views
  • 2 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Gilg, Deploying an AutoML pipeline in production while using a shared cluster in Databricks can be a bit tricky due to compatibility constraints. Let’s explore some potential workarounds: Shared Cluster with AutoML Compatibility: As you ment...

  • 0 kudos
1 More Replies
Brad
by Contributor
  • 2127 Views
  • 2 replies
  • 1 kudos

Resolved! What is the behavior when merge key is not unique

Hi, When using the MERGE statement, if merge key is not unique on both source and target, it will throw error. If merge key is unique in source but not unique in target, WHEN MATCHED THEN DELETE/UPDATE should work or not? For example merge key is id....

  • 2127 Views
  • 2 replies
  • 1 kudos
Latest Reply
Brad
Contributor
  • 1 kudos

Cool, this is what I tested out. Great to get confirmed. Thanks. BTW, https://medium.com/@ritik20023/delta-lake-upserting-without-primary-key-f4a931576b0 has a workaround which can fix the merge with duplicate merge key on both source and target.

  • 1 kudos
1 More Replies
Erik_L
by Contributor II
  • 596 Views
  • 2 replies
  • 1 kudos

Visualizations failing to show

I have a SQL query that generates a table. I created a visualization from that table with the UI. I then have a widget that updates a value used in the query and re-runs the SQL, but then the visualization shows nothing, that there is "1 row," but if...

Screenshot from 2024-04-05 10-23-03.png
  • 596 Views
  • 2 replies
  • 1 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 1 kudos

Hi @Erik_L , It seems like you’re encountering an issue with your visualization in Databricks. Let’s troubleshoot this! Here are a few common reasons why visualizations might not display as expected: Data Issues: Ensure that your SQL query is cor...

  • 1 kudos
1 More Replies
aurora
by New Contributor
  • 904 Views
  • 1 replies
  • 0 kudos

JDBC drivers for Microsoft Dataverse IO

I want to run Databricks ETLs on on-prem Unix, Azure and on AWS (in future). I am trying to find suitable JDBC drivers but couldn't find anything except CDATA which is very costly.Can someone please help me? Also, what could be other viable solutions...

Data Engineering
dataverse
JDBC
spark
  • 904 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @aurora, Let’s explore your options for running Databricks ETLs and connecting to Microsoft Dataverse. 1. JDBC Drivers for Databricks: Databricks provides its own JDBC drivers that allow you to connect to various data sources, including Microso...

  • 0 kudos
DumbBeaver
by New Contributor II
  • 561 Views
  • 1 replies
  • 0 kudos

Issue while writing data to unity catalog using JDBC

While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data. Driver used: com.databricks:databricks-jdbc:2.6.36Lets say I have the table has rows:+-+-+ |a|b| +-+-+ |1|2| |3|4| and I am appendi...

Data Engineering
JDBC
spark
Unity Catalog
  • 561 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @DumbBeaver, When writing data to a pre-existing table in the Unity Catalog using JDBC, it’s essential to understand how the .union operation and the .overwrite mode work. Union Operation: When you use .union to append rows to an existing Data...

  • 0 kudos
himanshu_k
by New Contributor
  • 646 Views
  • 1 replies
  • 0 kudos

Clarification Needed: Ensuring Correct Pagination with Offset and Limit in PySpark

Hi community,I hope you're all doing well. I'm currently engaged in a PySpark project where I'm implementing pagination-like functionality using the offset and limit functions. My aim is to retrieve data between a specified starting_index and ending_...

  • 646 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @himanshu_k, Let’s delve into your questions regarding pagination using the offset and limit functions in PySpark, especially when dealing with partitioned data frames. Consistency of offset and limit Functions: The offset and limit functions ...

  • 0 kudos
Leszek
by Contributor
  • 540 Views
  • 1 replies
  • 0 kudos

[Delta Sharing - open sharing protocol] Token rotation

Hi, Do you have any experience of rotating Tokens in Delta Sharing automatically?There is an option to do that using CLI (Create and manage data recipients for Delta Sharing | Databricks on AWS). But what to do next? Sending new link to the token via...

  • 540 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Leszek, Rotating tokens in Delta Sharing is a crucial security practice. Let’s break down the steps: Token Rotation: First, you’ve already taken the right step by using the CLI to create and manage data recipients for Delta Sharing. When you...

  • 0 kudos
Check
by New Contributor
  • 868 Views
  • 1 replies
  • 0 kudos

How to call azure databricks api from azure api management

Hi,Has anyone successfully configure azure apim to access databricks rest api ? If yes, appreciate  he can provide the setup guide for me as I am stuck at this point.  Thanks.

Check_0-1712215875654.png
  • 868 Views
  • 1 replies
  • 0 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 0 kudos

Hi @Check, Configuring Azure API Management (APIM) to access Databricks REST API can be a bit tricky, but I’ll guide you through some potential approaches: Using Environment Variables and cURL: To execute Databricks API via a curl request, you ne...

  • 0 kudos
397973
by New Contributor III
  • 960 Views
  • 3 replies
  • 0 kudos

Having trouble installing my own Python wheel?

I want to install my own Python wheel package on a cluster but can't get it working. I tried two ways: I followed these steps: https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#:~:text=March%2025%2C%202024,code%...

Data Engineering
cluster
Notebook
  • 960 Views
  • 3 replies
  • 0 kudos
Latest Reply
shan_chandra
Esteemed Contributor
  • 0 kudos

@397973 - Once you uploaded the .whl file, did you had a chance to list the file manually in the notebook?  Also, did you had a chance to move the files to /Volumes .whl file?  

  • 0 kudos
2 More Replies
SyedSaqib
by New Contributor II
  • 889 Views
  • 2 replies
  • 0 kudos

Delta Live Table : [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view

Hi,I have a delta live table workflow with storage enabled for cloud storage to a blob store.Syntax of bronze table in notebook===@dlt.table(spark_conf = {"spark.databricks.delta.schema.autoMerge.enabled": "true"},table_properties = {"quality": "bron...

  • 889 Views
  • 2 replies
  • 0 kudos
Latest Reply
SyedSaqib
New Contributor II
  • 0 kudos

Hi Kaniz,Thanks for replying back.I am using python for delta live table creation, so how can I set these configurations?When creating the table, add the IF NOT EXISTS clause to tolerate pre-existing objects.consider using the OR REFRESH clause Answe...

  • 0 kudos
1 More Replies
Henrique_Lino
by New Contributor II
  • 1609 Views
  • 6 replies
  • 0 kudos

value is null after loading a saved df when using specific type in schema

 I am facing an issue when using databricks, when I set a specific type in my schema and read a json, its values are fine, but after saving my df and loading again, the value is gone.I have this sample code that shows this issue: from pyspark.sql.typ...

  • 1609 Views
  • 6 replies
  • 0 kudos
Latest Reply
Lakshay
Esteemed Contributor
  • 0 kudos

@Henrique_Lino , Where are you saving your df?

  • 0 kudos
5 More Replies
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!

Labels