cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Hal
by • New Contributor II
  • 1235 Views
  • 1 replies
  • 3 kudos

Connecting Power BI on Azure to Databricks on AWS?

Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?

Data Engineering
Reply
  • 1235 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
bhanadi
New Contributor II
  • 3 kudos

Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?

  • 3 kudos
Saurabh707344
by • New Contributor III
  • 17807 Views
  • 1 replies
  • 1 kudos

Platform and Approach Comparison

Do anyone have structure and crisp comparison between benefits of performing MLOps using below ways and what are the strong areas of each platform:a) Standalone Databricks where all pipelines and orchestration done on Databricks and external third pa...

Data Engineering
Reply
  • 17807 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 1 kudos

Hi @Saurabh Singh​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 1 kudos
pvignesh92
by • Honored Contributor
  • 1545 Views
  • 1 replies
  • 3 kudos

lnkd.in

Databricks Auto Loader is an interesting feature that can be used to load data incrementally.✳ It can process new data files as they arrive in the cloud object stores✳ It can be used to ingest JSON, CSV, PARQUET, AVRO, ORC, TEXT and even Binary file ...

Data Engineering
Reply
  • 1545 Views
  • 1 replies
  • 3 kudos
View Replies
Latest Reply
Ajay-Pandey
Esteemed Contributor III
  • 3 kudos

Thanks for sharing

  • 3 kudos
DianGermishuiz1
by • New Contributor III
  • 2624 Views
  • 4 replies
  • 0 kudos

What is the purpose of the databricks-workspace-stack-lambdazipsbucket S3 bucket created on Databricks AWS Account Provisioning?

An S3 bucket with the prefix "databricks-workspace-stack-lambdazipsbucket" was created by default when I created my AWS Databricks account. It is set to public access. It has one zip file in it called "lambda.zip". What is the purpose of this S3 buck...

Data Engineering
Reply
  • 2624 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 0 kudos

Hi @Dian Germishuizen​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best ans...

  • 0 kudos
3 More Replies
MShee
by • New Contributor II
  • 1660 Views
  • 1 replies
  • 1 kudos

Resolved! Unable to type in dropdown of dbutils.widgets.dropdown() In the AWS databricks widgets.dropdown, I'm unable to type input in the dropdown box like in the below screenshot:

image
Data Engineering
Reply
  • 1660 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
NandiniN
Databricks Employee
  • 1 kudos

Hello @M Shee​ ,In a drop down you can select a value from a list of provided values, not type the values in. What you might be interested in is a combobox - It is combination of text and dropdown. It allows to select a value from a provided list or ...

  • 1 kudos
JordanYaker
by • Contributor
  • 2023 Views
  • 1 replies
  • 1 kudos

What is the maximum number of workspaces per account using Databricks on AWS?

I've been looking through the documentation and I swear this used to be listed somewhere, but for the life of me I can't find it anymore.

Data Engineering
Reply
  • 2023 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
JordanYaker
Contributor
  • 1 kudos

Thanks @Kaniz Fatma​ 

  • 1 kudos
Ajay-Pandey
by • Esteemed Contributor III
  • 2220 Views
  • 1 replies
  • 6 kudos

Variable explorer in Databricks With Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open t...

Variable explorer in DatabricksWith Databricks Runtime 12.1 and above, you can directly observe current Python variables in the notebook UI.To open the variable explorer, click in the right sidebar. The variable explorer opens, showing the value and ...

image Untitled
Data Engineering
Reply
  • 2220 Views
  • 1 replies
  • 6 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 6 kudos

Thank you for sharing

  • 6 kudos
sonalitotade
by • New Contributor II
  • 1797 Views
  • 2 replies
  • 0 kudos

Capture events such as Start, Stop and Terminate of cluster.

Hi,I am using databricks with AWS.I need to capture events such as Start, Stop and Terminate of cluster and perform some other action based on the events that happened on the cluster.Is there a way I can achieve this in databricks?

Data Engineering
Reply
  • 1797 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
sonalitotade
New Contributor II
  • 0 kudos

Hi Daniel, thanks for the responseI would like to know if we can capture the event logs as shown in the image below when an event occurs on the cluster.

  • 0 kudos
1 More Replies
rams
by • Contributor
  • 2227 Views
  • 3 replies
  • 4 kudos

Resolved! 14 day trial version console showing blank screen after login

I have taken a trial version of Databricks and wanted to configure it with AWS. but after login it was showing as blank screen since 20 hours. can someone help me with this. Note: strictly i have to use AWS with Databricks for configuration.

Data Engineering
Reply
  • 2227 Views
  • 3 replies
  • 4 kudos
View Replies
Latest Reply
Aviral-Bhardwaj
Esteemed Contributor III
  • 4 kudos

try to reach your account manager

  • 4 kudos
2 More Replies
Manimkm08
by • New Contributor III
  • 2638 Views
  • 3 replies
  • 0 kudos

Jobs are failed with AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE

We have assigned 3 dedicated subnets (one per AZ ) to the Databricks workspace each with /24 CIDR but noticed that all the jobs are running into a single subnet which causes AWS_INSUFFICIENT_FREE_ADDRESSES_IN_SUBNET_FAILURE.Is there a way to segregat...

Data Engineering
Reply
  • 2638 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
Manimkm08
New Contributor III
  • 0 kudos

@karthik p​ Have configured one subnet per AZ(total 3). Have followed the same steps as mentioned in the document. Is there a way to check whether the Databricks uses all the subnets or not?@Debayan Mukherjee​ am not getting how to use LB in this set...

  • 0 kudos
2 More Replies
augustin
by • New Contributor II
  • 4665 Views
  • 5 replies
  • 5 kudos

Mount an uncrypted AWS EFS in AWS Databricks

Hi,I want to mount an uncrypted AWS EFS in AWS Databricks. When I do:mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-abcdef.efs.region.amazonaws.com:/ /mnt/efs-uncryptedI get this error:mount.nfs4: moun...

Data Engineering
Reply
  • 4665 Views
  • 5 replies
  • 5 kudos
View Replies
Latest Reply
Andrei_Radulesc
Contributor III
  • 5 kudos

"To support NFS under LXC, some of the apparmor protections need to be lifted." (see https://theorangeone.net/posts/mount-nfs-inside-lxc/)

  • 5 kudos
4 More Replies
PaulP
by • New Contributor II
  • 2520 Views
  • 3 replies
  • 6 kudos

What is the best expected starting time for a cluster when using a pool?

Hi! I'm doing some tests to get an idea of how much time could be saved starting a cluster by using a pool and was wondering if the results I get are what should be expected.We're using AWS Databricks and used i3.xlarge as instance type (if that matt...

Data Engineering
Reply
  • 2520 Views
  • 3 replies
  • 6 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 6 kudos

Hi @Paul Pelletier​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...

  • 6 kudos
2 More Replies
Manimkm08
by • New Contributor III
  • 4761 Views
  • 4 replies
  • 4 kudos

Bootstrap Timeout during cluster start on AWS cloud

Sometimes am getting the below error when the cluster is started. Have attached the system log of the below mentioned instance from AWS. In recent days am getting this error for very frequently. Have seen same error is reported earlier and it marked ...

Data Engineering
Reply
  • 4761 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Manimkm08
New Contributor III
  • 4 kudos

@Kaniz Fatma​ @karthik p​  Since morning, am facing the issue again. Seems the issue is intermittent and it fails the pipeline in mid of the ETL process. Couldn't able to get the exact root cause of the issue. Can someone provide what would be the wo...

  • 4 kudos
3 More Replies
swetha
by • New Contributor III
  • 5905 Views
  • 2 replies
  • 2 kudos

Databricks job cluster logs

I am using databricks job cluster for multitask jobs, when my job failed/succeeded I couldn't see any logs, Do I need to add any location in advanced options, cluster logging to see the logs for the failed/succeeded jobs or what it is and how it work...

Data Engineering
Reply
  • 5905 Views
  • 2 replies
  • 2 kudos
View Replies
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @swetha kadiyala​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Th...

  • 2 kudos
1 More Replies
fundat
by • New Contributor II
  • 3420 Views
  • 3 replies
  • 1 kudos

Metastore - Access validation failed

Hi,When i create a metastore in aws databricks, i always have this error in the picture bellow.Eventhought i follow this link https://docs.databricks.com/data-governance/unity-catalog/get-started.html#cloud-tenant-setup-aws

metastore_databricks_error
Data Engineering
Reply
  • 3420 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
karthik_p
Esteemed Contributor
  • 1 kudos

@Anatole Cadet​ looks your IAM role is not properly configured, can you please check

  • 1 kudos
2 More Replies
Labels