cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

joao_vnb
by New Contributor III
  • 5616 Views
  • 2 replies
  • 0 kudos
  • 5616 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @joao vnb​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we c...

  • 0 kudos
1 More Replies
Thanapat_S
by Contributor
  • 3155 Views
  • 2 replies
  • 0 kudos

Resolved! Table access control is deprecated in Databricks Runtime for Machine Learning

After reviewing this Deprecations, I discovered that Table Access Control is not supported in Databricks Runtime for Machine Learning.I want to understand why table access control is not designed for ML runtime. Is there any reason behind this?

image.png
  • 3155 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Thanapat Sontayasara​ Table Access Control (TAC) is a feature in Databricks that allows you to restrict access to specific tables in your workspace based on user or group identity.According to the Databricks documentation, TAC is not supported in th...

  • 0 kudos
1 More Replies
Prashant777
by New Contributor II
  • 1820 Views
  • 2 replies
  • 0 kudos

Not able to merge Databricks cluster delta tables(location Abfs) with PowerBi for analytics Purpose

Hi everyone .. i have an issue while connecting azure  Databricks cluster delta tables(location Abfs) with PowerBi for analytics Purpose. but after succesfully linking cluster using Server Hostname and HTTP Path with powerbi is showing an error:-Data...

Error
  • 1820 Views
  • 2 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

Did you use the partner connect?Makes the connection pretty easy.

  • 0 kudos
1 More Replies
kll
by New Contributor III
  • 7828 Views
  • 2 replies
  • 0 kudos

AttributeError: module 'pyspark.dbutils' has no attribute 'fs'

`AttributeError` when attempting to transfer files from `dbfs` filestore in DataBricks to a local directory.  import pyspark.dbutils as pdbutils   pdbutils.fs.cp("/dbfs/Data/file1.csv", "/Users/Downloads/")   Traceback (most recent call last):   ...

  • 7828 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Keval Shah​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...

  • 0 kudos
1 More Replies
CDICSteph
by New Contributor
  • 2444 Views
  • 2 replies
  • 0 kudos

Need pattern for loading a million small XML files

Hi, looking for the right solution pattern for this scenario: We have millions of relatively small XML files (currently sitting in ADLS) that we have to load into delta lake. Each XML file has to be read, parsed, and pivoted before writing to a delta...

  • 2444 Views
  • 2 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Steph Swierenga​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answe...

  • 0 kudos
1 More Replies
Krish1
by New Contributor II
  • 8899 Views
  • 4 replies
  • 0 kudos

Error while mounting ADLS in python using AccountKey

I'm using the below code using Account key to mount ADLS in python but running into error:shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.IllegalArgumentException: The String is not a valid Base64-encoded string. Can you pleas...

  • 8899 Views
  • 4 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Krish Lam​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers you...

  • 0 kudos
3 More Replies
Tripalink
by New Contributor III
  • 4285 Views
  • 6 replies
  • 2 kudos

Resolved! Auto-Suggestion Completes with Return, want only Tab

How do I adjust the settings for the auto-suggestion? If I click Return, then it fills in the suggestion. If I click Tab, then it fills in the suggestion. I would really like it to only use the auto-suggestion value when I click Tab. How can I change...

  • 4285 Views
  • 6 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Dagart Allison​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answer...

  • 2 kudos
5 More Replies
kinsun
by New Contributor II
  • 3813 Views
  • 2 replies
  • 3 kudos

Resolved! Azure Key Vault Keys client library for Python - keys list permission issue

Dear Databricks ExpertI am trying to get a key which is stored in the Azure Key Vault, using Azure Key Vault Keys client library for Python. However error was met.Python Code:#from azure.identity import DefaultAzureCredentialfrom azure.identity impor...

  • 3813 Views
  • 2 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

Hi @KS LAU​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your q...

  • 3 kudos
1 More Replies
AnuVat
by New Contributor III
  • 39043 Views
  • 7 replies
  • 13 kudos

Resolved! How to read data from a table into a dataframe outside of Databricks environment?

Hi, I am working on an ML project and I need to access the data in tables hosted in my Databricks cluster through a notebook that I am running locally. This has been very easy while I run the notebooks in Databricks but I cannot figure out how to do ...

  • 39043 Views
  • 7 replies
  • 13 kudos
Latest Reply
chakri
New Contributor III
  • 13 kudos

We can use Apis and pyodbc to achieve this. Once go through the official documentation of databricks that might be helpful to access outside of the databricks environment.

  • 13 kudos
6 More Replies
sensanjoy
by Contributor
  • 4570 Views
  • 3 replies
  • 1 kudos

Resolved! Loading data from dataframe to Azure Storage Queue/Message Queue.

Hi Experts,We do have one use case where we have batch load that create a dataframe at end and now we want to load this data at Azure Storage Queue/Message Queue so that some Rest API can read the data/messages from the queue later and process it acc...

  • 4570 Views
  • 3 replies
  • 1 kudos
Latest Reply
sensanjoy
Contributor
  • 1 kudos

@Suteja Kanuri​  looking for your input here. Thanks.

  • 1 kudos
2 More Replies
arlok
by New Contributor
  • 6121 Views
  • 4 replies
  • 1 kudos

Partner session - Data Engineer Associate Course Schedule help

I have been wanting to enroll into the Data Engineer Associate Course as my co is a Databricks partner. I have been unsuccessful thusfar whenever the session has happened in North America and have always been waitlisted.1.Is there a hack to get a slo...

  • 6121 Views
  • 4 replies
  • 1 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 1 kudos

Hi @Lokesh AR​,Just a friendly follow-up. Do you still need help? Please let us know if you still need help

  • 1 kudos
3 More Replies
SrinuM
by New Contributor III
  • 1228 Views
  • 1 replies
  • 0 kudos

Not ab;e to login into databricks community edition

I forgot my password and when i am trying to reset my password , it was struck at reset password page.Could you please help me with it.

  • 1228 Views
  • 1 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

Adding @Vidula Khanna​ and @Kaniz Fatma​ for visibility to help with this request

  • 0 kudos
jasperputs
by New Contributor III
  • 12509 Views
  • 5 replies
  • 0 kudos

Databricks SQL Dashboard refresh not updating

I am trying to create a SQL Dashboard on top of a streaming dataset (Delta format). I created multiple queries referencing the file on the datalake, not a hive table. With these queries I created multiple visualizations in a Databricks SQL Dashboard....

image.png image
  • 12509 Views
  • 5 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

@Jasper Puts​ :Here are some potential solutions or next steps you can try:Check the query refresh rate: Confirm that the queries are also set to refresh every minute or less. If the queries are not refreshing at the same rate as the dashboard, this ...

  • 0 kudos
4 More Replies
MarsSu
by New Contributor II
  • 9201 Views
  • 5 replies
  • 1 kudos

Resolved! Databricks job about spark structured streaming zero downtime deployment in terraform.

I would like to ask how to implement zero downtime deployment of spark structured streaming in databricks job compute with terraform. Because we will upgrade spark application code version. But currently we found every deployment will cancel original...

  • 9201 Views
  • 5 replies
  • 1 kudos
Latest Reply
Anonymous
Not applicable
  • 1 kudos

@Mars Su​ :Yes, you can implement zero downtime deployment of Spark Structured Streaming in Databricks job compute using Terraform. One way to achieve this is by using Databricks' "job clusters" feature, which allows you to create a cluster specifica...

  • 1 kudos
4 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels